[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:08:56 dist-test-slave-2dw5 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:08:56 dist-test-slave-2dw5 krb5kdc[399](info): set up 2 sockets
Sep 26 13:08:56 dist-test-slave-2dw5 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 13:08:57.087153 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:08:57 dist-test-slave-2dw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:08:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:08:57.209676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46691
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:08:57.289527 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:08:57.289770 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:08:57.289803 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:08:57.291081 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:08:57.291132 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:08:57.291157 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:08:57.291173 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:08:57.291188 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:08:57.292780 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:08:57.293064 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:08:57.293325 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:08:57.298833 412 server_base.cc:1034] running on GCE node
W20260926 13:08:57.298944 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:08:57.298982 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:08:57.299008 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:08:57.299372 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:08:57.299675 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:08:57.300835 412 hybrid_clock.cc:648] HybridClock initialized: now 1790428137300795 us; error 51 us; skew 500 ppm
Sep 26 13:08:57 dist-test-slave-2dw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:08:57.303514 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:08:57.303966 412 webserver.cc:460] Webserver started at http://127.0.98.62:34013/ using document root <none> and password file <none>
I20260926 13:08:57.304221 412 fs_manager.cc:359] Metadata directory not provided
I20260926 13:08:57.304267 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:08:57.304373 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:08:57.305437 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "63adbb2e212a4e24a6a4dca4f5a5674a"
format_stamp: "Formatted at 2026-09-26 13:08:57 on dist-test-slave-2dw5"
server_key: "3f28fff5760ba1e189c020d69a64f3de"
server_key_iv: "b4b0f7fd516816ab977e6993a446c0ee"
server_key_version: "encryptionkey@0"
I20260926 13:08:57.305765 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "63adbb2e212a4e24a6a4dca4f5a5674a"
format_stamp: "Formatted at 2026-09-26 13:08:57 on dist-test-slave-2dw5"
server_key: "3f28fff5760ba1e189c020d69a64f3de"
server_key_iv: "b4b0f7fd516816ab977e6993a446c0ee"
server_key_version: "encryptionkey@0"
I20260926 13:08:57.307186 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:08:57.308066 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:08:57.308267 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:08:57.308370 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "63adbb2e212a4e24a6a4dca4f5a5674a"
format_stamp: "Formatted at 2026-09-26 13:08:57 on dist-test-slave-2dw5"
server_key: "3f28fff5760ba1e189c020d69a64f3de"
server_key_iv: "b4b0f7fd516816ab977e6993a446c0ee"
server_key_version: "encryptionkey@0"
I20260926 13:08:57.308431 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:08:57.321379 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:08:57.323642 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:08:57.323767 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:08:57.327476 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46691
I20260926 13:08:57.327521 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46691 every 8 connection(s)
I20260926 13:08:57.327865 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:08:57.328792 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:08:57.331041 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a: Bootstrap starting.
I20260926 13:08:57.331569 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a: Neither blocks nor log segments found. Creating new log.
I20260926 13:08:57.331806 480 log.cc:824] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:08:57.332544 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a: No bootstrap required, opened a new log
I20260926 13:08:57.334045 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63adbb2e212a4e24a6a4dca4f5a5674a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46691 } }
I20260926 13:08:57.334204 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:08:57.334225 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63adbb2e212a4e24a6a4dca4f5a5674a, State: Initialized, Role: FOLLOWER
I20260926 13:08:57.334333 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63adbb2e212a4e24a6a4dca4f5a5674a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46691 } }
I20260926 13:08:57.334380 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:08:57.334419 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:08:57.334465 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:08:57.334782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 412
I20260926 13:08:57.334918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:08:57.335196 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63adbb2e212a4e24a6a4dca4f5a5674a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46691 } }
I20260926 13:08:57.335319 480 leader_election.cc:302] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63adbb2e212a4e24a6a4dca4f5a5674a; no voters:
I20260926 13:08:57.335479 480 leader_election.cc:288] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:08:57.335631 484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:08:57.335704 392 external_mini_cluster.cc:1428] Setting key 1502d5df5c218bcba3ea0afcb04ed9f4
I20260926 13:08:57.335748 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:08:57.335822 484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [term 1 LEADER]: Becoming Leader. State: Replica: 63adbb2e212a4e24a6a4dca4f5a5674a, State: Running, Role: LEADER
I20260926 13:08:57.335911 484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63adbb2e212a4e24a6a4dca4f5a5674a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46691 } }
I20260926 13:08:57.336287 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63adbb2e212a4e24a6a4dca4f5a5674a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63adbb2e212a4e24a6a4dca4f5a5674a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46691 } } }
I20260926 13:08:57.336350 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [sys.catalog]: This master's current role is: LEADER
I20260926 13:08:57.336468 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63adbb2e212a4e24a6a4dca4f5a5674a. Latest consensus state: current_term: 1 leader_uuid: "63adbb2e212a4e24a6a4dca4f5a5674a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63adbb2e212a4e24a6a4dca4f5a5674a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46691 } } }
I20260926 13:08:57.336509 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a [sys.catalog]: This master's current role is: LEADER
I20260926 13:08:57.339046 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:08:57.339558 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:08:57.342537 492 catalog_manager.cc:1347] Generated new cluster ID: 4010bfccaf3a4892a3cca5effdf74558
I20260926 13:08:57.342654 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:08:57 dist-test-slave-2dw5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:08:57.352767 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:08:57.353371 492 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:08:57.354522 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:08:57.337094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.0.98.62:46691)
0926 13:08:57.337321 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 13:08:57.337331 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:08:57.337518 (+ 187us) server_negotiation.cc:407] Connection header received
0926 13:08:57.339306 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:08:57.339312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:08:57.339372 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:08:57.340628 (+ 1256us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:08:57.342067 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:08:57.343461 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:08:57.345057 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:08:57.345711 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:08:57.345801 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:08:57.350406 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:08:57.350412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:08:57.350412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:08:57.350425 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:08:57.352499 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:08:57.352995 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:08:57.352999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:08:57.352999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:08:57.353106 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:08:57.353370 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:08:57.353371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:08:57.353371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:08:57.353688 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:08:57.353825 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 13:08:57.354117 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 13:08:57.354269 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260926 13:08:57.360014 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63adbb2e212a4e24a6a4dca4f5a5674a: Generated new TSK 0
I20260926 13:08:57.360216 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260926 13:08:58.302356 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.938s user 0.000s sys 0.002s
I20260926 13:08:58.302788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46691
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:08:58.393316 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:08:58.393548 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:08:58.393572 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:08:58.394760 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:08:58.394809 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:08:58.394861 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:08:58.396450 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:08:58.396668 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:08:58.396860 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:08:58.402902 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:08:58.402944 506 server_base.cc:1034] running on GCE node
W20260926 13:08:58.402930 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:08:58.402921 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:08:58.403512 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:08:58.403782 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:08:58.404971 506 hybrid_clock.cc:648] HybridClock initialized: now 1790428138404952 us; error 46 us; skew 500 ppm
Sep 26 13:08:58 dist-test-slave-2dw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:08:58.408313 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:08:58.408948 506 webserver.cc:460] Webserver started at http://127.0.98.1:35567/ using document root <none> and password file <none>
I20260926 13:08:58.409169 506 fs_manager.cc:359] Metadata directory not provided
I20260926 13:08:58.409221 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:08:58.409349 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:08:58.410461 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "38d04a7c13374850860353cf37f8e5be"
format_stamp: "Formatted at 2026-09-26 13:08:58 on dist-test-slave-2dw5"
server_key: "56ac4053521bef8744eadb30b1cd705d"
server_key_iv: "d9f3423f98c3a1ef21809cf4830f1c5b"
server_key_version: "encryptionkey@0"
I20260926 13:08:58.410938 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "38d04a7c13374850860353cf37f8e5be"
format_stamp: "Formatted at 2026-09-26 13:08:58 on dist-test-slave-2dw5"
server_key: "56ac4053521bef8744eadb30b1cd705d"
server_key_iv: "d9f3423f98c3a1ef21809cf4830f1c5b"
server_key_version: "encryptionkey@0"
I20260926 13:08:58.412513 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:08:58.413730 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:08:58.414103 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:08:58.414222 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "38d04a7c13374850860353cf37f8e5be"
format_stamp: "Formatted at 2026-09-26 13:08:58 on dist-test-slave-2dw5"
server_key: "56ac4053521bef8744eadb30b1cd705d"
server_key_iv: "d9f3423f98c3a1ef21809cf4830f1c5b"
server_key_version: "encryptionkey@0"
I20260926 13:08:58.414311 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:08:58.427150 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:08:58.429524 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:08:58.429790 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:08:58.430059 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:08:58.430395 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:08:58.430434 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:08:58.430469 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:08:58.430490 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:08:58.438300 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37053
I20260926 13:08:58.438424 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37053 every 8 connection(s)
I20260926 13:08:58.438777 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:08:58.439100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 506
I20260926 13:08:58.439196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:08:58.439366 392 external_mini_cluster.cc:1428] Setting key 7c866a797831c5ad6ec0f11a9be75a77
Sep 26 13:08:58 dist-test-slave-2dw5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:08:58.450397 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:08:58.440166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59849 (local address 127.0.98.62:46691)
0926 13:08:58.440285 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 13:08:58.440287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:08:58.440301 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:08:58.440869 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:08:58.440870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:08:58.440886 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:08:58.440979 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:08:58.442442 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:08:58.443213 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:08:58.444447 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:08:58.444661 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:08:58.444729 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:08:58.447319 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:08:58.447321 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:08:58.447321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:08:58.447331 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:08:58.448918 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:08:58.449346 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:08:58.449348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:08:58.449348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:08:58.449416 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:08:58.449657 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:08:58.449658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:08:58.449658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:08:58.449791 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:08:58.449857 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:08:58.450190 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 13:08:58.450288 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260926 13:08:58.450558 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:08:58.439676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46691 (local address 127.0.98.1:59849)
0926 13:08:58.439891 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 13:08:58.439992 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0926 13:08:58.440730 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:08:58.441072 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:08:58.441078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:08:58.441964 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:08:58.442300 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:08:58.442308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:08:58.443356 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:08:58.443358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:08:58.444349 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:08:58.444353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:08:58.444745 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:08:58.444746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:08:58.444882 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:08:58.445504 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:08:58.445513 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:08:58.447200 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:08:58.449009 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:08:58.449018 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:08:58.449021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:08:58.449270 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:08:58.449477 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:08:58.449478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:08:58.449478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:08:58.449596 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:08:58.449855 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:08:58.449862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:08:58.450031 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 13:08:58.450145 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 13:08:58.450324 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
I20260926 13:08:58.451299 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46691
I20260926 13:08:58.451440 635 heartbeater.cc:464] Registering TS with master...
I20260926 13:08:58.451694 635 heartbeater.cc:511] Master 127.0.98.62:46691 requested a full tablet report, sending...
I20260926 13:08:58.452397 444 ts_manager.cc:194] Registered new tserver with Master: 38d04a7c13374850860353cf37f8e5be (127.0.98.1:37053)
I20260926 13:08:58.453080 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260926 13:08:58.643836 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.203s user 0.002s sys 0.001s
I20260926 13:08:58.644299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46691
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:08:58.735762 641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:08:58.736069 641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:08:58.736109 641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:08:58.737464 641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:08:58.737514 641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:08:58.737561 641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:08:58.739195 641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:08:58.739462 641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:08:58.739699 641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:08:58.745370 649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:08:58.745370 646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:08:58.745417 647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:08:58.745608 641 server_base.cc:1034] running on GCE node
I20260926 13:08:58.746009 641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:08:58.746292 641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:08:58.747454 641 hybrid_clock.cc:648] HybridClock initialized: now 1790428138747400 us; error 82 us; skew 500 ppm
Sep 26 13:08:58 dist-test-slave-2dw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:08:58.750128 641 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:08:58.750602 641 webserver.cc:460] Webserver started at http://127.0.98.2:41693/ using document root <none> and password file <none>
I20260926 13:08:58.750864 641 fs_manager.cc:359] Metadata directory not provided
I20260926 13:08:58.750926 641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:08:58.751070 641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:08:58.752238 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "22412896d37a42f4997a02d509b4520b"
format_stamp: "Formatted at 2026-09-26 13:08:58 on dist-test-slave-2dw5"
server_key: "5e59ba13761ac2c94207f978cc357343"
server_key_iv: "1e8ea7842d0d46ce6341556ffcf7bd89"
server_key_version: "encryptionkey@0"
I20260926 13:08:58.752601 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "22412896d37a42f4997a02d509b4520b"
format_stamp: "Formatted at 2026-09-26 13:08:58 on dist-test-slave-2dw5"
server_key: "5e59ba13761ac2c94207f978cc357343"
server_key_iv: "1e8ea7842d0d46ce6341556ffcf7bd89"
server_key_version: "encryptionkey@0"
I20260926 13:08:58.754055 641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:08:58.755009 656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:08:58.755264 641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:08:58.755378 641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "22412896d37a42f4997a02d509b4520b"
format_stamp: "Formatted at 2026-09-26 13:08:58 on dist-test-slave-2dw5"
server_key: "5e59ba13761ac2c94207f978cc357343"
server_key_iv: "1e8ea7842d0d46ce6341556ffcf7bd89"
server_key_version: "encryptionkey@0"
I20260926 13:08:58.755448 641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:08:58.781819 641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:08:58.784340 641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:08:58.784505 641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:08:58.784840 641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:08:58.785178 641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:08:58.785213 641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:08:58.785243 641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:08:58.785259 641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:08:58.792153 641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42307
I20260926 13:08:58.792237 769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42307 every 8 connection(s)
I20260926 13:08:58.792562 641 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:08:58 dist-test-slave-2dw5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:08:58.800370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 641
I20260926 13:08:58.800483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:08:58.800652 392 external_mini_cluster.cc:1428] Setting key 747390395c30e8e3682dd352e61f5969
I20260926 13:08:58.804059 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:08:58.793422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34105 (local address 127.0.98.62:46691)
0926 13:08:58.793532 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 13:08:58.793535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:08:58.793777 (+ 242us) server_negotiation.cc:407] Connection header received
0926 13:08:58.794391 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:08:58.794392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:08:58.794407 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:08:58.794470 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:08:58.795615 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:08:58.796257 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:08:58.797384 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:08:58.797601 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:08:58.797668 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:08:58.800454 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:08:58.800456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:08:58.800456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:08:58.800468 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:08:58.802080 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:08:58.802776 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:08:58.802778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:08:58.802778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:08:58.802872 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:08:58.803184 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:08:58.803186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:08:58.803186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:08:58.803322 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:08:58.803429 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 13:08:58.803824 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 13:08:58.803938 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:08:58.804153 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:08:58.793362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46691 (local address 127.0.98.2:34105)
0926 13:08:58.793612 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 13:08:58.793660 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:08:58.794276 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:08:58.794562 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:08:58.794566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:08:58.795259 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:08:58.795506 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:08:58.795512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:08:58.796356 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:08:58.796357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:08:58.797292 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:08:58.797296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:08:58.797673 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:08:58.797673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:08:58.797817 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:08:58.798418 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:08:58.798429 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:08:58.800275 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:08:58.802181 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:08:58.802192 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:08:58.802196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:08:58.802662 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:08:58.802945 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:08:58.802946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:08:58.802946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:08:58.803091 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:08:58.803431 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:08:58.803438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:08:58.803667 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 13:08:58.803772 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 13:08:58.803946 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260926 13:08:58.804919 770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46691
I20260926 13:08:58.805066 770 heartbeater.cc:464] Registering TS with master...
I20260926 13:08:58.805291 770 heartbeater.cc:511] Master 127.0.98.62:46691 requested a full tablet report, sending...
I20260926 13:08:58.805766 444 ts_manager.cc:194] Registered new tserver with Master: 22412896d37a42f4997a02d509b4520b (127.0.98.2:42307)
I20260926 13:08:58.806188 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34105
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:08:58.830920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46691
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:08:58.919343 775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:08:58.919569 775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:08:58.919593 775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:08:58.920802 775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:08:58.920857 775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:08:58.920885 775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:08:58.922444 775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46691
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:08:58.922701 775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:08:58.922976 775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:08:58.930095 780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:08:58.930081 781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:08:58.930088 783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:08:58.930259 775 server_base.cc:1034] running on GCE node
I20260926 13:08:58.930639 775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:08:58.930965 775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:08:58.932148 775 hybrid_clock.cc:648] HybridClock initialized: now 1790428138932134 us; error 34 us; skew 500 ppm
Sep 26 13:08:58 dist-test-slave-2dw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:08:58.935209 775 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:08:58.935753 775 webserver.cc:460] Webserver started at http://127.0.98.3:36601/ using document root <none> and password file <none>
I20260926 13:08:58.935977 775 fs_manager.cc:359] Metadata directory not provided
I20260926 13:08:58.936025 775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:08:58.936148 775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:08:58.937521 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "679cbc14a81a40a6bd3571f82a9a1001"
format_stamp: "Formatted at 2026-09-26 13:08:58 on dist-test-slave-2dw5"
server_key: "8c7b684444978eb87f98f44ef00e20b0"
server_key_iv: "31366e5c4eca173003395c9e2ad30a32"
server_key_version: "encryptionkey@0"
I20260926 13:08:58.937866 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "679cbc14a81a40a6bd3571f82a9a1001"
format_stamp: "Formatted at 2026-09-26 13:08:58 on dist-test-slave-2dw5"
server_key: "8c7b684444978eb87f98f44ef00e20b0"
server_key_iv: "31366e5c4eca173003395c9e2ad30a32"
server_key_version: "encryptionkey@0"
I20260926 13:08:58.939421 775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:08:58.940476 790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:08:58.940706 775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:08:58.940834 775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "679cbc14a81a40a6bd3571f82a9a1001"
format_stamp: "Formatted at 2026-09-26 13:08:58 on dist-test-slave-2dw5"
server_key: "8c7b684444978eb87f98f44ef00e20b0"
server_key_iv: "31366e5c4eca173003395c9e2ad30a32"
server_key_version: "encryptionkey@0"
I20260926 13:08:58.940915 775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:08:58.973165 775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:08:58.975780 775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:08:58.975919 775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:08:58.976202 775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:08:58.976536 775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:08:58.976572 775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:08:58.976599 775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:08:58.976632 775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:08:58.984972 775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43561
I20260926 13:08:58.985064 903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43561 every 8 connection(s)
I20260926 13:08:58.985419 775 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:08:58.987048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 775
I20260926 13:08:58.987157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:08:58.987305 392 external_mini_cluster.cc:1428] Setting key a651426e6ebda49255b2de64da240a9a
Sep 26 13:08:58 dist-test-slave-2dw5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:08:58.995968 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:08:58.986470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50575 (local address 127.0.98.62:46691)
0926 13:08:58.986574 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 13:08:58.986576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:08:58.986718 (+ 142us) server_negotiation.cc:407] Connection header received
0926 13:08:58.987492 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:08:58.987494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:08:58.987507 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:08:58.987579 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:08:58.988742 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:08:58.989296 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:08:58.990387 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:08:58.990586 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:08:58.990646 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:08:58.993053 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:08:58.993055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:08:58.993055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:08:58.993063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:08:58.994553 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:08:58.995016 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:08:58.995017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:08:58.995017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:08:58.995086 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:08:58.995311 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:08:58.995312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:08:58.995312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:08:58.995442 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:08:58.995493 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:08:58.995806 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 13:08:58.995889 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 13:08:58.996043 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:08:58.986320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46691 (local address 127.0.98.3:50575)
0926 13:08:58.986558 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 13:08:58.986615 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 13:08:58.987353 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:08:58.987674 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:08:58.987678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:08:58.988377 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:08:58.988633 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:08:58.988639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:08:58.989391 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:08:58.989391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:08:58.990310 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:08:58.990312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:08:58.990690 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:08:58.990691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:08:58.990814 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:08:58.991439 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:08:58.991449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:08:58.992949 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:08:58.994654 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:08:58.994660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:08:58.994662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:08:58.994942 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:08:58.995148 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:08:58.995149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:08:58.995149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:08:58.995246 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:08:58.995493 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:08:58.995496 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:08:58.995620 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 13:08:58.995703 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 13:08:58.995831 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":92,"threads_started":1}
I20260926 13:08:58.996767 904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46691
I20260926 13:08:58.996869 904 heartbeater.cc:464] Registering TS with master...
I20260926 13:08:58.997074 904 heartbeater.cc:511] Master 127.0.98.62:46691 requested a full tablet report, sending...
I20260926 13:08:58.997553 444 ts_manager.cc:194] Registered new tserver with Master: 679cbc14a81a40a6bd3571f82a9a1001 (127.0.98.3:43561)
I20260926 13:08:58.997988 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50575
I20260926 13:08:58.999642 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:08:59 dist-test-slave-2dw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:08:59 dist-test-slave-2dw5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:08:59.025581 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:08:59.016792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.0.98.1:37053)
0926 13:08:59.016951 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 13:08:59.016958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:08:59.016974 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:08:59.017046 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:08:59.017049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:08:59.017068 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:08:59.017139 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:08:59.017843 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:08:59.018637 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:08:59.019468 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:08:59.019825 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:08:59.019891 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:08:59.022526 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:08:59.022537 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:08:59.022538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:08:59.022546 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:08:59.024277 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:08:59.024728 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:08:59.024731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:08:59.024732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:08:59.024806 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:08:59.024983 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:08:59.024983 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:08:59.024984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:08:59.025150 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:08:59.025326 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 13:08:59.025371 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 13:08:59.025456 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
W20260926 13:08:59.026050 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55928
Sep 26 13:08:59 dist-test-slave-2dw5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:08:59 dist-test-slave-2dw5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:08:59.052191 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:08:59.043558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.0.98.1:37053)
0926 13:08:59.043647 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 13:08:59.043650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:08:59.043789 (+ 139us) server_negotiation.cc:407] Connection header received
0926 13:08:59.043905 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:08:59.043907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:08:59.043918 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:08:59.043966 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:08:59.044685 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:08:59.045286 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:08:59.046150 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:08:59.046353 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:08:59.046438 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:08:59.049467 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:08:59.049469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:08:59.049470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:08:59.049478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:08:59.050976 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:08:59.051376 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:08:59.051378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:08:59.051378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:08:59.051446 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:08:59.051693 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:08:59.051694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:08:59.051694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:08:59.051811 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:08:59.051901 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:08:59.052013 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 13:08:59.052109 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:08:59.052870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 506
I20260926 13:08:59.057965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 641
I20260926 13:08:59.064590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 775
I20260926 13:08:59.069531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 412
2026-09-26T13:08:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2962 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:08:59 dist-test-slave-2dw5 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:08:59 dist-test-slave-2dw5 krb5kdc[918](info): set up 2 sockets
Sep 26 13:08:59 dist-test-slave-2dw5 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260926 13:09:00.030510 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:00 dist-test-slave-2dw5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:09:00.149802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35983
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:00.235183 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:00.235410 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:00.235445 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:00.236670 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:00.236729 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:00.236750 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:00.236769 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:00.236789 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:00.238325 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:00.238590 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:00.238818 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:00.244007 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:00.244072 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:00.244068 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:00.244518 931 server_base.cc:1034] running on GCE node
I20260926 13:09:00.244766 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:00.245065 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:00.246232 931 hybrid_clock.cc:648] HybridClock initialized: now 1790428140246208 us; error 41 us; skew 500 ppm
Sep 26 13:09:00 dist-test-slave-2dw5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:00.249003 931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:09:00.249488 931 webserver.cc:460] Webserver started at http://127.0.98.62:38163/ using document root <none> and password file <none>
I20260926 13:09:00.249711 931 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:00.249759 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:00.249890 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:00.250991 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "fcf296f2e2904449b0a9095fc15df1b2"
format_stamp: "Formatted at 2026-09-26 13:09:00 on dist-test-slave-2dw5"
server_key: "84b29f561e865132da952ae3f2a10e69"
server_key_iv: "00e39cacc3d84afa48ff6f9c08270981"
server_key_version: "encryptionkey@0"
I20260926 13:09:00.251324 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "fcf296f2e2904449b0a9095fc15df1b2"
format_stamp: "Formatted at 2026-09-26 13:09:00 on dist-test-slave-2dw5"
server_key: "84b29f561e865132da952ae3f2a10e69"
server_key_iv: "00e39cacc3d84afa48ff6f9c08270981"
server_key_version: "encryptionkey@0"
I20260926 13:09:00.252636 931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:00.253475 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:00.253695 931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:00.253798 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "fcf296f2e2904449b0a9095fc15df1b2"
format_stamp: "Formatted at 2026-09-26 13:09:00 on dist-test-slave-2dw5"
server_key: "84b29f561e865132da952ae3f2a10e69"
server_key_iv: "00e39cacc3d84afa48ff6f9c08270981"
server_key_version: "encryptionkey@0"
I20260926 13:09:00.253865 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:00.282373 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:00.285147 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:00.285306 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:00.289309 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35983
I20260926 13:09:00.289376 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35983 every 8 connection(s)
I20260926 13:09:00.289721 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:00.290678 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:00.293079 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2: Bootstrap starting.
I20260926 13:09:00.293655 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:00.293921 999 log.cc:824] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:00.294611 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2: No bootstrap required, opened a new log
I20260926 13:09:00.295405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 931
I20260926 13:09:00.295483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:00.295646 392 external_mini_cluster.cc:1428] Setting key ae98b57c34ac7b18f0bf00c9d88b2443
I20260926 13:09:00.296767 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf296f2e2904449b0a9095fc15df1b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35983 } }
I20260926 13:09:00.296940 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:00.296988 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf296f2e2904449b0a9095fc15df1b2, State: Initialized, Role: FOLLOWER
I20260926 13:09:00.297114 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf296f2e2904449b0a9095fc15df1b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35983 } }
I20260926 13:09:00.297201 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:00.297238 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:00.297286 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:00.298401 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf296f2e2904449b0a9095fc15df1b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35983 } }
I20260926 13:09:00.298511 999 leader_election.cc:302] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcf296f2e2904449b0a9095fc15df1b2; no voters:
I20260926 13:09:00.298681 999 leader_election.cc:288] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:00.298744 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:00.298904 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [term 1 LEADER]: Becoming Leader. State: Replica: fcf296f2e2904449b0a9095fc15df1b2, State: Running, Role: LEADER
I20260926 13:09:00.298926 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:00.299000 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf296f2e2904449b0a9095fc15df1b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35983 } }
I20260926 13:09:00.299316 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcf296f2e2904449b0a9095fc15df1b2. Latest consensus state: current_term: 1 leader_uuid: "fcf296f2e2904449b0a9095fc15df1b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf296f2e2904449b0a9095fc15df1b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35983 } } }
I20260926 13:09:00.299403 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:00.299327 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcf296f2e2904449b0a9095fc15df1b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcf296f2e2904449b0a9095fc15df1b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35983 } } }
I20260926 13:09:00.299544 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:00.299616 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:00.300112 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:00.302134 1011 catalog_manager.cc:1347] Generated new cluster ID: fb348d5a3f9847909f527307af2166ca
I20260926 13:09:00.302191 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:09:00 dist-test-slave-2dw5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:00.306734 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.296218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.0.98.62:35983)
0926 13:09:00.296413 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:00.296423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:00.296469 (+ 46us) server_negotiation.cc:407] Connection header received
0926 13:09:00.296726 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:00.296734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:00.296796 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:00.296938 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:00.297651 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.299085 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.300195 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.300817 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.301284 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.303470 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:00.303474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:00.303474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:00.303485 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:00.305002 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:00.305330 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:00.305333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:00.305333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:00.305434 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:00.305614 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:00.305615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:00.305615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:00.305849 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:00.306018 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:00.306259 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 13:09:00.306521 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260926 13:09:00.306798 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:00.307513 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:00.316949 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcf296f2e2904449b0a9095fc15df1b2: Generated new TSK 0
I20260926 13:09:00.317162 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:09:00.351989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35983
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:00.445026 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:00.445257 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:00.445283 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:00.446537 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:00.446588 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:00.446615 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:00.448122 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35983
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:00.448352 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:00.448566 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:00.453584 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:00.453591 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:00.453585 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:00.453635 1025 server_base.cc:1034] running on GCE node
I20260926 13:09:00.453903 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:00.454124 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:00.455271 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790428140455251 us; error 36 us; skew 500 ppm
Sep 26 13:09:00 dist-test-slave-2dw5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:00.457754 1025 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:09:00.458263 1025 webserver.cc:460] Webserver started at http://127.0.98.1:37283/ using document root <none> and password file <none>
I20260926 13:09:00.458480 1025 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:00.458531 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:00.458666 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:00.459877 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "a55ea9ce301f4816ae512495505f6f89"
format_stamp: "Formatted at 2026-09-26 13:09:00 on dist-test-slave-2dw5"
server_key: "0e1d421263e27e61b300a969ef297c2c"
server_key_iv: "365b08d64843f8091a93bf9b1407c3a6"
server_key_version: "encryptionkey@0"
I20260926 13:09:00.460249 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a55ea9ce301f4816ae512495505f6f89"
format_stamp: "Formatted at 2026-09-26 13:09:00 on dist-test-slave-2dw5"
server_key: "0e1d421263e27e61b300a969ef297c2c"
server_key_iv: "365b08d64843f8091a93bf9b1407c3a6"
server_key_version: "encryptionkey@0"
I20260926 13:09:00.461665 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:00.462675 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:00.462922 1025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:00.463024 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "a55ea9ce301f4816ae512495505f6f89"
format_stamp: "Formatted at 2026-09-26 13:09:00 on dist-test-slave-2dw5"
server_key: "0e1d421263e27e61b300a969ef297c2c"
server_key_iv: "365b08d64843f8091a93bf9b1407c3a6"
server_key_version: "encryptionkey@0"
I20260926 13:09:00.463112 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:00.477543 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:00.479890 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:00.480041 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:00.480278 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:00.480620 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:00.480666 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:00.480696 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:00.480712 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:00.486509 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45305
I20260926 13:09:00.486553 1153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45305 every 8 connection(s)
I20260926 13:09:00.486888 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:09:00.487474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 1025
I20260926 13:09:00.487562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:00.487741 392 external_mini_cluster.cc:1428] Setting key 2437683849c8544b992a8343c5035606
Sep 26 13:09:00 dist-test-slave-2dw5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:09:00.497213 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.487683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50665 (local address 127.0.98.62:35983)
0926 13:09:00.487736 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:00.487738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:00.487968 (+ 230us) server_negotiation.cc:407] Connection header received
0926 13:09:00.488514 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:00.488515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:00.488529 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:00.488587 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:00.489654 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.490141 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.491619 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.491833 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.491898 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.494420 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:00.494422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:00.494422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:00.494433 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:00.495834 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:00.496286 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:00.496286 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:00.496287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:00.496333 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:00.496538 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:00.496539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:00.496539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:00.496650 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:00.496704 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:00.497071 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 13:09:00.497143 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:09:00.497545 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.487687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35983 (local address 127.0.98.1:50665)
0926 13:09:00.487835 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:00.487877 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:00.488422 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:00.488659 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:00.488663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:00.489265 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:00.489554 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.489561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.490238 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.490239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.491495 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.491499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.491925 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.491926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.492076 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.492611 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:00.492622 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:00.494303 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:00.495912 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:00.495918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:00.495921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:00.496226 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:00.496374 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:00.496375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:00.496375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:00.496486 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:00.496730 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:00.496736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:00.496866 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 13:09:00.497050 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 13:09:00.497217 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 13:09:00.498091 1154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35983
I20260926 13:09:00.498193 1154 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:00.498385 1154 heartbeater.cc:511] Master 127.0.98.62:35983 requested a full tablet report, sending...
I20260926 13:09:00.498929 963 ts_manager.cc:194] Registered new tserver with Master: a55ea9ce301f4816ae512495505f6f89 (127.0.98.1:45305)
I20260926 13:09:00.499688 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:09:00.514428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35983
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:00.597754 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:00.597992 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:00.598027 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:00.599340 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:00.599386 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:00.599426 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:00.601392 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35983
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:00.601688 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:00.601913 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:00.606745 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:00.606755 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:00.606910 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:00.607016 1159 server_base.cc:1034] running on GCE node
I20260926 13:09:00.607223 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:00.607425 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:00.608587 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790428140608595 us; error 50 us; skew 500 ppm
Sep 26 13:09:00 dist-test-slave-2dw5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:00.611081 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:09:00.611544 1159 webserver.cc:460] Webserver started at http://127.0.98.2:41825/ using document root <none> and password file <none>
I20260926 13:09:00.611745 1159 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:00.611793 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:00.611915 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:00.612962 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "f3cee4fba92944fe8e39354048b1a19d"
format_stamp: "Formatted at 2026-09-26 13:09:00 on dist-test-slave-2dw5"
server_key: "de1e3bbb997f33c90cee3e7803773887"
server_key_iv: "b1badb9d7d57e04b444618c08f0b0d71"
server_key_version: "encryptionkey@0"
I20260926 13:09:00.613286 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f3cee4fba92944fe8e39354048b1a19d"
format_stamp: "Formatted at 2026-09-26 13:09:00 on dist-test-slave-2dw5"
server_key: "de1e3bbb997f33c90cee3e7803773887"
server_key_iv: "b1badb9d7d57e04b444618c08f0b0d71"
server_key_version: "encryptionkey@0"
I20260926 13:09:00.614698 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:00.615552 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:00.615711 1159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:00.615794 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "f3cee4fba92944fe8e39354048b1a19d"
format_stamp: "Formatted at 2026-09-26 13:09:00 on dist-test-slave-2dw5"
server_key: "de1e3bbb997f33c90cee3e7803773887"
server_key_iv: "b1badb9d7d57e04b444618c08f0b0d71"
server_key_version: "encryptionkey@0"
I20260926 13:09:00.615861 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:00.624666 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:00.626928 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:00.627045 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:00.627223 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:00.627496 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:00.627526 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:00.627553 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:00.627584 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:00.633077 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38761
I20260926 13:09:00.633138 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38761 every 8 connection(s)
I20260926 13:09:00.633440 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:09:00.639359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 1159
I20260926 13:09:00.639436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
Sep 26 13:09:00 dist-test-slave-2dw5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:00.639581 392 external_mini_cluster.cc:1428] Setting key f4341191b35519e326c41452295d12ad
I20260926 13:09:00.642875 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.634185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58271 (local address 127.0.98.62:35983)
0926 13:09:00.634254 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:00.634256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:00.634421 (+ 165us) server_negotiation.cc:407] Connection header received
0926 13:09:00.634941 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:00.634943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:00.634952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:00.634983 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:00.635926 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.636458 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.637498 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.637713 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.637766 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.640148 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:00.640149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:00.640150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:00.640157 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:00.641537 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:00.641980 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:00.641981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:00.641982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:00.642033 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:00.642248 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:00.642249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:00.642250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:00.642355 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:00.642409 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:00.642664 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 13:09:00.642748 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:09:00.643059 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.634151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35983 (local address 127.0.98.2:58271)
0926 13:09:00.634311 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:00.634348 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:00.634818 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:00.635086 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:00.635089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:00.635648 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:00.635838 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.635842 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.636572 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.636573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.637414 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.637417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.637792 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.637793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.637903 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.638493 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:00.638501 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:00.640038 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:00.641660 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:00.641679 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:00.641682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:00.641909 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:00.642088 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:00.642089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:00.642089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:00.642190 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:00.642410 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:00.642416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:00.642547 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 13:09:00.642636 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:09:00.642800 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260926 13:09:00.643455 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35983
I20260926 13:09:00.643554 1288 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:00.643729 1288 heartbeater.cc:511] Master 127.0.98.62:35983 requested a full tablet report, sending...
I20260926 13:09:00.644075 963 ts_manager.cc:194] Registered new tserver with Master: f3cee4fba92944fe8e39354048b1a19d (127.0.98.2:38761)
I20260926 13:09:00.644603 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58271
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:09:00.666309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35983
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:00.749590 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:00.749835 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:00.749864 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:00.751222 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:00.751279 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:00.751310 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:00.752823 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35983
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:00.753046 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:00.753235 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:00.758517 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:00.758540 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:00.758592 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:00.758741 1293 server_base.cc:1034] running on GCE node
I20260926 13:09:00.758996 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:00.759227 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:00.760382 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790428140760365 us; error 33 us; skew 500 ppm
Sep 26 13:09:00 dist-test-slave-2dw5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:00.763183 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:09:00.763634 1293 webserver.cc:460] Webserver started at http://127.0.98.3:41557/ using document root <none> and password file <none>
I20260926 13:09:00.763839 1293 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:00.763885 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:00.763999 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:00.765048 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "78dbf926429f4c30b4574cb887d1cefe"
format_stamp: "Formatted at 2026-09-26 13:09:00 on dist-test-slave-2dw5"
server_key: "4784258104b715643e3fa284d6ed5444"
server_key_iv: "8da913b3c272c830298dca5383f6f245"
server_key_version: "encryptionkey@0"
I20260926 13:09:00.765358 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "78dbf926429f4c30b4574cb887d1cefe"
format_stamp: "Formatted at 2026-09-26 13:09:00 on dist-test-slave-2dw5"
server_key: "4784258104b715643e3fa284d6ed5444"
server_key_iv: "8da913b3c272c830298dca5383f6f245"
server_key_version: "encryptionkey@0"
I20260926 13:09:00.766705 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:09:00.767527 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:00.767696 1293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:09:00.767777 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "78dbf926429f4c30b4574cb887d1cefe"
format_stamp: "Formatted at 2026-09-26 13:09:00 on dist-test-slave-2dw5"
server_key: "4784258104b715643e3fa284d6ed5444"
server_key_iv: "8da913b3c272c830298dca5383f6f245"
server_key_version: "encryptionkey@0"
I20260926 13:09:00.767840 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:00.784358 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:00.786936 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:00.787091 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:00.787352 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:00.787724 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:00.787770 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:00.787801 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:00.787825 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:00.793732 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35135
I20260926 13:09:00.793781 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35135 every 8 connection(s)
I20260926 13:09:00.794093 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:09:00 dist-test-slave-2dw5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:00.801293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 1293
I20260926 13:09:00.801371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:00.801723 392 external_mini_cluster.cc:1428] Setting key 6dae0fab2e9d3f4e141588aefcc77e6e
I20260926 13:09:00.803779 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.794729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39067 (local address 127.0.98.62:35983)
0926 13:09:00.794787 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:00.794789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:00.795032 (+ 243us) server_negotiation.cc:407] Connection header received
0926 13:09:00.795497 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:00.795498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:00.795507 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:00.795609 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:00.796529 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.797040 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.798126 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.798336 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.798394 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.800601 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:00.800603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:00.800603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:00.800610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:00.802268 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:00.802760 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:00.802761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:00.802761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:00.802816 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:00.803117 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:00.803118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:00.803118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:00.803236 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:00.803297 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:00.803640 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 13:09:00.803714 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 13:09:00.803920 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.794718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35983 (local address 127.0.98.3:39067)
0926 13:09:00.794916 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:00.794954 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:00.795420 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:00.795668 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:00.795672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:00.796249 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:00.796434 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.796438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.797119 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.797119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.797970 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.797972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.798405 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.798406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.798528 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.799063 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:00.799071 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:00.800500 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:00.802356 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:00.802364 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:00.802367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:00.802685 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:00.802900 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:00.802901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:00.802901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:00.803003 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:00.803302 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:00.803306 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:00.803462 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 13:09:00.803590 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 13:09:00.803721 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
I20260926 13:09:00.804486 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35983
I20260926 13:09:00.804607 1422 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:00.804777 1422 heartbeater.cc:511] Master 127.0.98.62:35983 requested a full tablet report, sending...
I20260926 13:09:00.805142 963 ts_manager.cc:194] Registered new tserver with Master: 78dbf926429f4c30b4574cb887d1cefe (127.0.98.3:35135)
I20260926 13:09:00.805509 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39067
I20260926 13:09:00.813824 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:09:00.822201 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.815584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.0.98.62:35983)
0926 13:09:00.815656 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:00.815658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:00.815713 (+ 55us) server_negotiation.cc:407] Connection header received
0926 13:09:00.815767 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:00.815767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:00.815775 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:00.815827 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:00.816421 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.816948 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.817833 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.818106 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.818166 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.819686 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:00.819688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:00.819688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:00.819695 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:00.821134 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:00.821468 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:00.821469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:00.821470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:00.821559 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:00.821736 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:00.821737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:00.821737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:00.821891 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:00.821964 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:00.822069 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 13:09:00.822132 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:09:00.824618 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:00.826243 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:00.833120 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.828661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38761 (local address 127.0.0.1:42830)
0926 13:09:00.829093 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:00.829125 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:00.829189 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:00.829579 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:00.829579 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:00.829582 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:00.829632 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.829634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.830896 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.830896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.832075 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.832078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.832808 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.832809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.832900 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.832916 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 13:09:00.832975 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:09:00.833038 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":5,"thread_start_us":43,"threads_started":1}
I20260926 13:09:00.833127 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.828788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.0.98.2:38761)
0926 13:09:00.829420 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:00.829423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:00.829431 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:00.829471 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:00.829473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:00.829489 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:00.829567 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:00.829709 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.830759 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.832163 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.832734 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.832798 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.832861 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:00.832996 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 13:09:00.833035 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":88,"threads_started":1}
I20260926 13:09:00.833451 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.828788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.0.98.3:35135)
0926 13:09:00.829038 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:00.829042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:00.829335 (+ 293us) server_negotiation.cc:407] Connection header received
0926 13:09:00.829376 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:00.829378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:00.829391 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:00.829458 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:00.829733 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.830642 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.832240 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.832971 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.833051 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.833105 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:00.833314 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 13:09:00.833354 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":122,"threads_started":1}
I20260926 13:09:00.833462 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.828953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.0.98.1:45305)
0926 13:09:00.829189 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:00.829192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:00.829868 (+ 676us) server_negotiation.cc:407] Connection header received
0926 13:09:00.829919 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:00.829922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:00.830044 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:00.830120 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:00.830346 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.831245 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.832349 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.833057 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.833121 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.833170 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:00.833344 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 13:09:00.833391 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":70,"threads_started":1}
I20260926 13:09:00.833669 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.828663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35135 (local address 127.0.0.1:43648)
0926 13:09:00.829272 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:00.829279 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:00.829319 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:00.829528 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:00.829531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:00.829534 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:00.829625 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.829627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.830728 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.830730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.832112 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.832114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.833144 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.833145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.833211 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.833215 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:00.833555 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 13:09:00.833607 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":72,"threads_started":1}
I20260926 13:09:00.833431 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.828831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45305 (local address 127.0.0.1:41196)
0926 13:09:00.829667 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:00.829677 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:00.829725 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:00.830212 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:00.830213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:00.830215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:00.830262 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.830263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.831333 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.831334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.832266 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.832268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.833201 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.833202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.833252 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.833255 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:00.833307 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 13:09:00.833352 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260926 13:09:00.834263 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 5539b20ae99449098fb4839926a8ebb1 (DEFAULT_TABLE table=test-table [id=2d1326c0aab4400b9d6a47ba32312d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:00.834359 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 5539b20ae99449098fb4839926a8ebb1 (DEFAULT_TABLE table=test-table [id=2d1326c0aab4400b9d6a47ba32312d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:00.834453 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 5539b20ae99449098fb4839926a8ebb1 (DEFAULT_TABLE table=test-table [id=2d1326c0aab4400b9d6a47ba32312d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:00.834677 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5539b20ae99449098fb4839926a8ebb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:00.834682 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5539b20ae99449098fb4839926a8ebb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:00.834736 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5539b20ae99449098fb4839926a8ebb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:00.837512 1439 tablet_bootstrap.cc:492] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe: Bootstrap starting.
I20260926 13:09:00.838124 1441 tablet_bootstrap.cc:492] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89: Bootstrap starting.
I20260926 13:09:00.838479 1439 tablet_bootstrap.cc:654] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:00.838682 1441 tablet_bootstrap.cc:654] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:00.839003 1440 tablet_bootstrap.cc:492] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d: Bootstrap starting.
I20260926 13:09:00.839056 1439 log.cc:824] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:00.839228 1441 log.cc:824] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:00.839594 1440 tablet_bootstrap.cc:654] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:00.839912 1441 tablet_bootstrap.cc:492] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89: No bootstrap required, opened a new log
I20260926 13:09:00.839933 1439 tablet_bootstrap.cc:492] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe: No bootstrap required, opened a new log
I20260926 13:09:00.839980 1441 ts_tablet_manager.cc:1397] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 13:09:00.839994 1439 ts_tablet_manager.cc:1397] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:09:00.840109 1440 log.cc:824] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:00.840896 1440 tablet_bootstrap.cc:492] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d: No bootstrap required, opened a new log
I20260926 13:09:00.840991 1440 ts_tablet_manager.cc:1397] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:00.842155 1439 raft_consensus.cc:348] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3cee4fba92944fe8e39354048b1a19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "a55ea9ce301f4816ae512495505f6f89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45305 } } peers { permanent_uuid: "78dbf926429f4c30b4574cb887d1cefe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:09:00.842114 1441 raft_consensus.cc:348] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3cee4fba92944fe8e39354048b1a19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "a55ea9ce301f4816ae512495505f6f89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45305 } } peers { permanent_uuid: "78dbf926429f4c30b4574cb887d1cefe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:09:00.842295 1439 raft_consensus.cc:374] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:00.842298 1441 raft_consensus.cc:374] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:00.842327 1439 raft_consensus.cc:729] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78dbf926429f4c30b4574cb887d1cefe, State: Initialized, Role: FOLLOWER
I20260926 13:09:00.842334 1441 raft_consensus.cc:729] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55ea9ce301f4816ae512495505f6f89, State: Initialized, Role: FOLLOWER
I20260926 13:09:00.842468 1441 consensus_queue.cc:260] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3cee4fba92944fe8e39354048b1a19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "a55ea9ce301f4816ae512495505f6f89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45305 } } peers { permanent_uuid: "78dbf926429f4c30b4574cb887d1cefe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:09:00.842526 1439 consensus_queue.cc:260] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3cee4fba92944fe8e39354048b1a19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "a55ea9ce301f4816ae512495505f6f89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45305 } } peers { permanent_uuid: "78dbf926429f4c30b4574cb887d1cefe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:09:00.842811 1439 ts_tablet_manager.cc:1428] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:09:00.842890 1422 heartbeater.cc:503] Master 127.0.98.62:35983 was elected leader, sending a full tablet report...
I20260926 13:09:00.842770 1441 ts_tablet_manager.cc:1428] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 13:09:00.843077 1440 raft_consensus.cc:348] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3cee4fba92944fe8e39354048b1a19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "a55ea9ce301f4816ae512495505f6f89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45305 } } peers { permanent_uuid: "78dbf926429f4c30b4574cb887d1cefe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:09:00.843251 1440 raft_consensus.cc:374] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:00.843290 1440 raft_consensus.cc:729] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3cee4fba92944fe8e39354048b1a19d, State: Initialized, Role: FOLLOWER
I20260926 13:09:00.843286 1154 heartbeater.cc:503] Master 127.0.98.62:35983 was elected leader, sending a full tablet report...
I20260926 13:09:00.843397 1440 consensus_queue.cc:260] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3cee4fba92944fe8e39354048b1a19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "a55ea9ce301f4816ae512495505f6f89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45305 } } peers { permanent_uuid: "78dbf926429f4c30b4574cb887d1cefe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:09:00.843657 1440 ts_tablet_manager.cc:1428] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:09:00.843770 1288 heartbeater.cc:503] Master 127.0.98.62:35983 was elected leader, sending a full tablet report...
W20260926 13:09:00.884088 1289 tablet.cc:2369] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:09:00.947010 1445 raft_consensus.cc:482] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:00.947135 1445 raft_consensus.cc:504] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3cee4fba92944fe8e39354048b1a19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "a55ea9ce301f4816ae512495505f6f89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45305 } } peers { permanent_uuid: "78dbf926429f4c30b4574cb887d1cefe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:09:00.947440 1445 leader_election.cc:288] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3cee4fba92944fe8e39354048b1a19d (127.0.98.2:38761), 78dbf926429f4c30b4574cb887d1cefe (127.0.98.3:35135)
I20260926 13:09:00.950443 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.947731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34791 (local address 127.0.98.3:35135)
0926 13:09:00.947778 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:00.947780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:00.947787 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:09:00.947815 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:00.947816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:00.947828 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:00.947886 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:00.948050 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.948612 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.949685 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.950134 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.950185 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.950193 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:00.950361 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 13:09:00.950388 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:09:00.950469 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.947642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35135 (local address 127.0.98.1:34791)
0926 13:09:00.947702 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:00.947715 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:00.947762 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:00.947904 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:00.947904 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:00.947906 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:00.947984 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.947987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.948702 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.948703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.949570 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.949572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.950204 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.950205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.950293 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.950297 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:00.950354 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 13:09:00.950394 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 13:09:00.950834 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.947666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38761 (local address 127.0.98.1:46585)
0926 13:09:00.947851 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:00.947861 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:00.947927 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:00.948049 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:00.948049 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:00.948050 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:00.948191 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.948194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.948978 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.948979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.949808 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:00.949810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.950630 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.950631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:00.950699 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.950703 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:00.950754 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 13:09:00.950780 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260926 13:09:00.950905 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.947729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46585 (local address 127.0.98.2:38761)
0926 13:09:00.947772 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:00.947774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:00.947926 (+ 152us) server_negotiation.cc:407] Connection header received
0926 13:09:00.947974 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:00.947975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:00.947987 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:00.948038 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:00.948357 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.948900 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.949895 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.950429 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.950487 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.950496 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:00.950775 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 13:09:00.950818 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:09:00.950857 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5539b20ae99449098fb4839926a8ebb1" candidate_uuid: "a55ea9ce301f4816ae512495505f6f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78dbf926429f4c30b4574cb887d1cefe" is_pre_election: true
I20260926 13:09:00.951007 1376 raft_consensus.cc:2454] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a55ea9ce301f4816ae512495505f6f89 in term 0.
I20260926 13:09:00.951186 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5539b20ae99449098fb4839926a8ebb1" candidate_uuid: "a55ea9ce301f4816ae512495505f6f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3cee4fba92944fe8e39354048b1a19d" is_pre_election: true
I20260926 13:09:00.951215 1044 leader_election.cc:302] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78dbf926429f4c30b4574cb887d1cefe, a55ea9ce301f4816ae512495505f6f89; no voters:
I20260926 13:09:00.951315 1242 raft_consensus.cc:2454] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a55ea9ce301f4816ae512495505f6f89 in term 0.
I20260926 13:09:00.951385 1445 raft_consensus.cc:2781] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:00.951452 1445 raft_consensus.cc:482] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:00.951473 1445 raft_consensus.cc:3037] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:00.952154 1445 raft_consensus.cc:504] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3cee4fba92944fe8e39354048b1a19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "a55ea9ce301f4816ae512495505f6f89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45305 } } peers { permanent_uuid: "78dbf926429f4c30b4574cb887d1cefe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:09:00.952283 1445 leader_election.cc:288] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [CANDIDATE]: Term 1 election: Requested vote from peers f3cee4fba92944fe8e39354048b1a19d (127.0.98.2:38761), 78dbf926429f4c30b4574cb887d1cefe (127.0.98.3:35135)
I20260926 13:09:00.952421 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5539b20ae99449098fb4839926a8ebb1" candidate_uuid: "a55ea9ce301f4816ae512495505f6f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78dbf926429f4c30b4574cb887d1cefe"
I20260926 13:09:00.952469 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5539b20ae99449098fb4839926a8ebb1" candidate_uuid: "a55ea9ce301f4816ae512495505f6f89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3cee4fba92944fe8e39354048b1a19d"
I20260926 13:09:00.952531 1242 raft_consensus.cc:3037] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:00.952512 1376 raft_consensus.cc:3037] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:00.953238 1242 raft_consensus.cc:2454] T 5539b20ae99449098fb4839926a8ebb1 P f3cee4fba92944fe8e39354048b1a19d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a55ea9ce301f4816ae512495505f6f89 in term 1.
I20260926 13:09:00.953244 1376 raft_consensus.cc:2454] T 5539b20ae99449098fb4839926a8ebb1 P 78dbf926429f4c30b4574cb887d1cefe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a55ea9ce301f4816ae512495505f6f89 in term 1.
I20260926 13:09:00.953406 1042 leader_election.cc:302] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a55ea9ce301f4816ae512495505f6f89, f3cee4fba92944fe8e39354048b1a19d; no voters:
I20260926 13:09:00.953485 1445 raft_consensus.cc:2781] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:00.953634 1445 raft_consensus.cc:686] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [term 1 LEADER]: Becoming Leader. State: Replica: a55ea9ce301f4816ae512495505f6f89, State: Running, Role: LEADER
I20260926 13:09:00.953712 1445 consensus_queue.cc:237] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3cee4fba92944fe8e39354048b1a19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } } peers { permanent_uuid: "a55ea9ce301f4816ae512495505f6f89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45305 } } peers { permanent_uuid: "78dbf926429f4c30b4574cb887d1cefe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:09:00.954551 963 catalog_manager.cc:5495] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89 reported cstate change: term changed from 0 to 1, leader changed from <none> to a55ea9ce301f4816ae512495505f6f89 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a55ea9ce301f4816ae512495505f6f89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3cee4fba92944fe8e39354048b1a19d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a55ea9ce301f4816ae512495505f6f89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78dbf926429f4c30b4574cb887d1cefe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } health_report { overall_health: UNKNOWN } } }
Sep 26 13:09:00 dist-test-slave-2dw5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:09:00.983915 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:00.976329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.0.98.1:45305)
0926 13:09:00.976415 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:00.976417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:00.976485 (+ 68us) server_negotiation.cc:407] Connection header received
0926 13:09:00.976562 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:00.976563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:00.976576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:00.976629 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:00.977202 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.977724 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.978501 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:00.978693 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:00.978748 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:00.981056 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:00.981070 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:00.981071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:00.981079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:00.982769 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:00.983148 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:00.983152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:00.983152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:00.983211 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:00.983376 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:00.983377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:00.983377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:00.983526 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:00.983706 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:00.983742 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 13:09:00.983820 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 13:09:00.987653 1155 tablet.cc:2369] T 5539b20ae99449098fb4839926a8ebb1 P a55ea9ce301f4816ae512495505f6f89: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 13:09:00 dist-test-slave-2dw5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428140, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:09:00 dist-test-slave-2dw5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428140, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:09:01.007324 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:01.000233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.0.98.1:45305)
0926 13:09:01.000290 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:01.000292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:01.000376 (+ 84us) server_negotiation.cc:407] Connection header received
0926 13:09:01.000451 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:01.000452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:01.000464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:01.000520 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:01.001145 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:01.001643 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:01.002463 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:01.002678 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:01.002738 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:01.005110 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:01.005112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:01.005112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:01.005120 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:01.006241 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:01.006579 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:01.006580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:01.006580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:01.006632 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:01.006773 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:01.006773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:01.006773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:01.006908 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:01.007003 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:01.007143 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 13:09:01.007210 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 13:09:01 dist-test-slave-2dw5 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:09:01 dist-test-slave-2dw5 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:09:01.029992 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:01.022638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.0.98.1:45305)
0926 13:09:01.022709 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:01.022711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:01.022758 (+ 47us) server_negotiation.cc:407] Connection header received
0926 13:09:01.022810 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:01.022811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:01.022820 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:01.022904 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:01.023492 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:01.023956 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:01.024770 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:01.024973 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:01.025039 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:01.027447 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:01.027449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:01.027449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:01.027457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:01.028841 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:01.029278 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:01.029279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:01.029279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:01.029336 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:01.029515 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:01.029516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:01.029516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:01.029645 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:01.029740 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:01.029856 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 13:09:01.029920 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 13:09:01.030500 1068 tablet_service.cc:2908] Rejecting scan request for tablet 5539b20ae99449098fb4839926a8ebb1: Uninitialized: clean time has not yet been initialized
I20260926 13:09:01.031754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 1025
I20260926 13:09:01.035952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 1159
I20260926 13:09:01.040360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 1293
I20260926 13:09:01.044531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 931
2026-09-26T13:09:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1973 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:01 dist-test-slave-2dw5 krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:01 dist-test-slave-2dw5 krb5kdc[1463](info): set up 2 sockets
Sep 26 13:09:01 dist-test-slave-2dw5 krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260926 13:09:02.006490 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:02 dist-test-slave-2dw5 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:09:02.126668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46015
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:02.207983 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:02.208216 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:02.208240 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:02.209443 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:02.209487 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 13:09:02.209501 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:02.209512 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:02.209537 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:02.209553 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:02.211225 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46015
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:02.211478 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:02.211689 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:02.216764 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:02.216753 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:02.216928 1476 server_base.cc:1034] running on GCE node
W20260926 13:09:02.216754 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:02.217231 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:02.217559 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:02.218729 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790428142218711 us; error 31 us; skew 500 ppm
Sep 26 13:09:02 dist-test-slave-2dw5 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:02.228516 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:09:02.229010 1476 webserver.cc:460] Webserver started at http://127.0.98.62:42629/ using document root <none> and password file <none>
I20260926 13:09:02.229230 1476 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:02.229275 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:02.229404 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:02.230501 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "5679fb23842f4122b4e37738dd80c43d"
format_stamp: "Formatted at 2026-09-26 13:09:02 on dist-test-slave-2dw5"
server_key: "405b54d1d357d01e3e3d6f4f2068b24d"
server_key_iv: "6b9f716e6bf15aa488d724fe301e70f7"
server_key_version: "encryptionkey@0"
I20260926 13:09:02.230826 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "5679fb23842f4122b4e37738dd80c43d"
format_stamp: "Formatted at 2026-09-26 13:09:02 on dist-test-slave-2dw5"
server_key: "405b54d1d357d01e3e3d6f4f2068b24d"
server_key_iv: "6b9f716e6bf15aa488d724fe301e70f7"
server_key_version: "encryptionkey@0"
I20260926 13:09:02.232290 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:02.233009 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:02.233209 1476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:09:02.233300 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "5679fb23842f4122b4e37738dd80c43d"
format_stamp: "Formatted at 2026-09-26 13:09:02 on dist-test-slave-2dw5"
server_key: "405b54d1d357d01e3e3d6f4f2068b24d"
server_key_iv: "6b9f716e6bf15aa488d724fe301e70f7"
server_key_version: "encryptionkey@0"
I20260926 13:09:02.233371 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:02.242540 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:02.244717 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:02.244867 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:02.249563 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46015
I20260926 13:09:02.249616 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46015 every 8 connection(s)
I20260926 13:09:02.249956 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:02.250578 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:02.251469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 1476
I20260926 13:09:02.251547 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:02.251694 392 external_mini_cluster.cc:1428] Setting key 6a717efbf97dfa34141745650a429867
I20260926 13:09:02.253427 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d: Bootstrap starting.
I20260926 13:09:02.254036 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:02.254346 1554 log.cc:824] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:02.255168 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d: No bootstrap required, opened a new log
I20260926 13:09:02.257476 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5679fb23842f4122b4e37738dd80c43d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46015 } }
I20260926 13:09:02.257627 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:02.257647 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5679fb23842f4122b4e37738dd80c43d, State: Initialized, Role: FOLLOWER
I20260926 13:09:02.257766 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5679fb23842f4122b4e37738dd80c43d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46015 } }
I20260926 13:09:02.257844 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:02.257875 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:02.257920 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:02.258536 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5679fb23842f4122b4e37738dd80c43d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46015 } }
I20260926 13:09:02.258651 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5679fb23842f4122b4e37738dd80c43d; no voters:
I20260926 13:09:02.258858 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:02.258908 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [term 1 FOLLOWER]: Leader election won for term 1
Sep 26 13:09:02 dist-test-slave-2dw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:02.259064 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [term 1 LEADER]: Becoming Leader. State: Replica: 5679fb23842f4122b4e37738dd80c43d, State: Running, Role: LEADER
I20260926 13:09:02.259127 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:02.259158 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5679fb23842f4122b4e37738dd80c43d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46015 } }
I20260926 13:09:02.259553 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5679fb23842f4122b4e37738dd80c43d. Latest consensus state: current_term: 1 leader_uuid: "5679fb23842f4122b4e37738dd80c43d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5679fb23842f4122b4e37738dd80c43d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46015 } } }
I20260926 13:09:02.259632 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:02.259754 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5679fb23842f4122b4e37738dd80c43d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5679fb23842f4122b4e37738dd80c43d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46015 } } }
I20260926 13:09:02.259876 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:02.260241 1565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:02.260769 1565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:02.262454 1565 catalog_manager.cc:1347] Generated new cluster ID: cafff60842454f6d8fa922f4affbf5df
I20260926 13:09:02.262502 1565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:02.263510 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.252251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.0.98.62:46015)
0926 13:09:02.252438 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.252447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.252479 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:09:02.253145 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.253154 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.253219 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.253381 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:02.254017 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.255355 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.256117 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.256770 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.256886 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.259564 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:02.259569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:02.259569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:02.259583 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:02.261739 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.262181 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.262184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.262185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.262289 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.262555 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.262556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.262557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.262903 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:02.263038 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.263160 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.263303 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260926 13:09:02.269181 1565 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:02.269752 1565 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:02.272260 1565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d: Generated new TSK 0
I20260926 13:09:02.272475 1565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:09:02.301910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46015
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:02.388531 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:02.388756 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:02.388789 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:02.389998 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:02.390048 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:02.390076 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:09:02.390101 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:02.391743 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:02.392004 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:02.392227 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:02.397245 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:02.397311 1580 server_base.cc:1034] running on GCE node
W20260926 13:09:02.397374 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:02.397311 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:02.397681 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:02.397913 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:02.399057 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790428142399039 us; error 29 us; skew 500 ppm
Sep 26 13:09:02 dist-test-slave-2dw5 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:02.401688 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:09:02.402149 1580 webserver.cc:460] Webserver started at http://127.0.98.1:45439/ using document root <none> and password file <none>
I20260926 13:09:02.402390 1580 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:02.402439 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:02.402565 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:02.403767 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "ec3a2365c22d4f1fac0029034a354988"
format_stamp: "Formatted at 2026-09-26 13:09:02 on dist-test-slave-2dw5"
server_key: "72e4c42f910feea958441970ea50c9ec"
server_key_iv: "2dd4f50907697957b254c30acc4bac93"
server_key_version: "encryptionkey@0"
I20260926 13:09:02.404093 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ec3a2365c22d4f1fac0029034a354988"
format_stamp: "Formatted at 2026-09-26 13:09:02 on dist-test-slave-2dw5"
server_key: "72e4c42f910feea958441970ea50c9ec"
server_key_iv: "2dd4f50907697957b254c30acc4bac93"
server_key_version: "encryptionkey@0"
I20260926 13:09:02.405462 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:02.406162 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:02.406385 1580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:02.406476 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "ec3a2365c22d4f1fac0029034a354988"
format_stamp: "Formatted at 2026-09-26 13:09:02 on dist-test-slave-2dw5"
server_key: "72e4c42f910feea958441970ea50c9ec"
server_key_iv: "2dd4f50907697957b254c30acc4bac93"
server_key_version: "encryptionkey@0"
I20260926 13:09:02.406533 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:02.428393 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:02.430536 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:02.430657 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:02.431175 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:02.431223 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:02.431257 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:02.431417 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:09:02 dist-test-slave-2dw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:02.438896 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39615
I20260926 13:09:02.439070 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39615 every 8 connection(s)
I20260926 13:09:02.439313 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:09:02.441941 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.431736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58351 (local address 127.0.98.62:46015)
0926 13:09:02.431783 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.431786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.432044 (+ 258us) server_negotiation.cc:407] Connection header received
0926 13:09:02.432574 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.432575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.432590 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.432651 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:02.433473 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.434230 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.435854 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.436175 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.436248 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.439034 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:02.439036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:02.439036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:02.439046 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:02.440526 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.440974 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.440975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.440975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.441042 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.441270 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.441271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.441271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.441390 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:02.441450 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.441774 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.441860 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:09:02.442045 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.431720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46015 (local address 127.0.98.1:58351)
0926 13:09:02.431911 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:02.431952 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:02.432472 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:02.432725 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:02.432729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:02.433044 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:02.433337 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.433344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.434367 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.434368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.435737 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.435740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.436268 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.436269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.436415 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.437025 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:02.437034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:02.438918 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:02.440621 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:02.440627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:02.440629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:02.440890 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:02.441101 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:02.441101 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:02.441102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:02.441212 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:02.441487 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:02.441492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:02.441621 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 13:09:02.441722 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 13:09:02.441855 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260926 13:09:02.442654 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46015
I20260926 13:09:02.442777 1711 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:02.442983 1711 heartbeater.cc:511] Master 127.0.98.62:46015 requested a full tablet report, sending...
I20260926 13:09:02.443593 1507 ts_manager.cc:194] Registered new tserver with Master: ec3a2365c22d4f1fac0029034a354988 (127.0.98.1:39615)
I20260926 13:09:02.444242 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58351
I20260926 13:09:02.447373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 1580
I20260926 13:09:02.447466 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:02.447638 392 external_mini_cluster.cc:1428] Setting key 58ceee05bb25c483726e335ac07ae3c6
I20260926 13:09:02.449282 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.443686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55141 (local address 127.0.98.62:46015)
0926 13:09:02.443724 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.443726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.443881 (+ 155us) server_negotiation.cc:407] Connection header received
0926 13:09:02.443937 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.443938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.443948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.444000 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:02.444317 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.444890 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.445575 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.445784 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.445836 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.446897 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:02.446899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:02.446899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:02.446906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:02.448271 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.448616 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.448616 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.448617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.448671 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.448889 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.448890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.448890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.448986 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:02.449030 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.449161 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.449222 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:09:02.449344 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.443631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46015 (local address 127.0.98.1:55141)
0926 13:09:02.443818 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:02.443827 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:02.443883 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:02.444003 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:02.444004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:02.444177 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:02.444239 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.444241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.444979 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.444980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.445504 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.445506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.445843 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.445844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.445911 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.446471 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:02.446475 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:02.446787 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:02.448349 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:02.448350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:02.448350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:02.448550 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:02.448761 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:02.448761 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:02.448762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:02.448837 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:02.449029 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:02.449030 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:02.449074 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 13:09:02.449133 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:09:02.449235 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:09:02.473977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46015
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:02.555330 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:02.555572 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:02.555596 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:02.556808 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:02.556857 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:02.556877 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:09:02.556895 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:02.558455 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:02.558702 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:02.558969 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:09:02.564350 1720 server_base.cc:1034] running on GCE node
W20260926 13:09:02.564265 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:02.564358 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:02.564283 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:02.564718 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:02.564973 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:02.566154 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790428142566141 us; error 29 us; skew 500 ppm
Sep 26 13:09:02 dist-test-slave-2dw5 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:02.568928 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:09:02.569482 1720 webserver.cc:460] Webserver started at http://127.0.98.2:44059/ using document root <none> and password file <none>
I20260926 13:09:02.569689 1720 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:02.569736 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:02.569867 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:02.570978 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "3096ff9c453e4485b2d15b82b9e9bb83"
format_stamp: "Formatted at 2026-09-26 13:09:02 on dist-test-slave-2dw5"
server_key: "4aac3a7ba88d5a2eca2c6cf112ad2b15"
server_key_iv: "8e43eaffa1dc1d32e7b1ebceea35eef3"
server_key_version: "encryptionkey@0"
I20260926 13:09:02.571316 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3096ff9c453e4485b2d15b82b9e9bb83"
format_stamp: "Formatted at 2026-09-26 13:09:02 on dist-test-slave-2dw5"
server_key: "4aac3a7ba88d5a2eca2c6cf112ad2b15"
server_key_iv: "8e43eaffa1dc1d32e7b1ebceea35eef3"
server_key_version: "encryptionkey@0"
I20260926 13:09:02.572623 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:02.573447 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:02.573678 1720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:02.573763 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "3096ff9c453e4485b2d15b82b9e9bb83"
format_stamp: "Formatted at 2026-09-26 13:09:02 on dist-test-slave-2dw5"
server_key: "4aac3a7ba88d5a2eca2c6cf112ad2b15"
server_key_iv: "8e43eaffa1dc1d32e7b1ebceea35eef3"
server_key_version: "encryptionkey@0"
I20260926 13:09:02.573832 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:02.592357 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:02.594650 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:02.594807 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:02.595341 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:02.595451 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:02.595492 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:02.595517 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:09:02 dist-test-slave-2dw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:02.602314 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34795
I20260926 13:09:02.602383 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34795 every 8 connection(s)
I20260926 13:09:02.602712 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:09:02.605458 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.595879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43643 (local address 127.0.98.62:46015)
0926 13:09:02.595944 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.595946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.596139 (+ 193us) server_negotiation.cc:407] Connection header received
0926 13:09:02.596789 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.596791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.596804 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.596836 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:02.597812 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.598590 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.599765 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.600079 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.600164 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.602783 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:02.602785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:02.602786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:02.602796 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:02.604125 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.604590 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.604591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.604591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.604641 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.604842 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.604843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.604843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.604975 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:02.605039 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.605267 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.605331 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:09:02.605541 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.595851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46015 (local address 127.0.98.2:43643)
0926 13:09:02.596006 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:02.596045 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:02.596656 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:02.597041 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:02.597046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:02.597417 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:02.597703 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.597709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.598686 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.598687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.599667 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.599670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.600185 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.600186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.600332 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.600880 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:02.600888 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:02.602593 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:02.604283 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:02.604288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:02.604290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:02.604529 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:02.604679 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:02.604679 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:02.604679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:02.604762 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:02.605039 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:02.605043 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:02.605167 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 13:09:02.605249 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 13:09:02.605378 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260926 13:09:02.606010 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46015
I20260926 13:09:02.606101 1851 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:02.606268 1851 heartbeater.cc:511] Master 127.0.98.62:46015 requested a full tablet report, sending...
I20260926 13:09:02.606673 1508 ts_manager.cc:194] Registered new tserver with Master: 3096ff9c453e4485b2d15b82b9e9bb83 (127.0.98.2:34795)
I20260926 13:09:02.607133 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43643
I20260926 13:09:02.608783 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 1720
I20260926 13:09:02.608884 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:02.609063 392 external_mini_cluster.cc:1428] Setting key 6086105182a77004e00646db3887013f
I20260926 13:09:02.612654 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.607044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44323 (local address 127.0.98.62:46015)
0926 13:09:02.607092 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.607094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.607144 (+ 50us) server_negotiation.cc:407] Connection header received
0926 13:09:02.607215 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.607216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.607226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.607275 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:02.607605 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.608214 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.608942 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.609220 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.609284 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.610110 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:02.610111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:02.610111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:02.610118 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:02.611604 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.611948 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.611949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.611949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.611998 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.612190 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.612190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.612191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.612319 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:02.612403 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.612530 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.612593 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 13:09:02.612958 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.606978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46015 (local address 127.0.98.2:44323)
0926 13:09:02.607094 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:02.607104 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:02.607158 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:02.607277 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:02.607278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:02.607474 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:02.607531 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.607533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.608298 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.608299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.608834 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.608836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.609283 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.609285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.609355 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.609735 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:02.609739 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:02.610032 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:02.611679 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:02.611680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:02.611681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:02.611885 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:02.612047 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:02.612048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:02.612048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:02.612108 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:02.612398 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:02.612399 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:02.612444 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 13:09:02.612511 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:09:02.612607 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:09:02.633467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46015
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:02.713050 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:02.713268 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:02.713291 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:02.714452 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:02.714496 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:02.714515 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:09:02.714532 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:02.716073 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:02.716295 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:02.716513 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:02.721685 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:02.721685 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:02.721685 1860 server_base.cc:1034] running on GCE node
W20260926 13:09:02.721685 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:02.722133 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:02.722357 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:02.723513 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790428142723469 us; error 60 us; skew 500 ppm
Sep 26 13:09:02 dist-test-slave-2dw5 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:02.726082 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:09:02.726536 1860 webserver.cc:460] Webserver started at http://127.0.98.3:35797/ using document root <none> and password file <none>
I20260926 13:09:02.726747 1860 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:02.726794 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:02.726948 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:02.728072 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "1e7a4194970f477cab7e3a244e3e888a"
format_stamp: "Formatted at 2026-09-26 13:09:02 on dist-test-slave-2dw5"
server_key: "42c6887693dd558e04d4d7c91ac75bd9"
server_key_iv: "fdb663efbe6a795ada43daba1641e5b4"
server_key_version: "encryptionkey@0"
I20260926 13:09:02.728402 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1e7a4194970f477cab7e3a244e3e888a"
format_stamp: "Formatted at 2026-09-26 13:09:02 on dist-test-slave-2dw5"
server_key: "42c6887693dd558e04d4d7c91ac75bd9"
server_key_iv: "fdb663efbe6a795ada43daba1641e5b4"
server_key_version: "encryptionkey@0"
I20260926 13:09:02.729656 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:02.730460 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:02.730628 1860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:02.730700 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "1e7a4194970f477cab7e3a244e3e888a"
format_stamp: "Formatted at 2026-09-26 13:09:02 on dist-test-slave-2dw5"
server_key: "42c6887693dd558e04d4d7c91ac75bd9"
server_key_iv: "fdb663efbe6a795ada43daba1641e5b4"
server_key_version: "encryptionkey@0"
I20260926 13:09:02.730762 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:02.761198 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:02.763375 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:02.763491 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:02.764079 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:02.764120 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:02.764142 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:02.764156 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:02.770818 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42343
I20260926 13:09:02.770948 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42343 every 8 connection(s)
Sep 26 13:09:02 dist-test-slave-2dw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:02.771209 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:02.774158 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.764482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40309 (local address 127.0.98.62:46015)
0926 13:09:02.764543 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.764545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.764772 (+ 227us) server_negotiation.cc:407] Connection header received
0926 13:09:02.765449 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.765449 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.765460 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.765514 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:02.766379 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.767091 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.768761 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.769025 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.769085 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.771638 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:02.771640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:02.771640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:02.771648 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:02.772964 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.773334 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.773335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.773335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.773380 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.773588 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.773589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.773589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.773716 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:02.773806 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.774026 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.774090 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:09:02.774353 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.764471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46015 (local address 127.0.98.3:40309)
0926 13:09:02.764632 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:02.764681 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:02.765341 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:02.765597 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:02.765602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:02.765964 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:02.766286 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.766292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.767340 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.767341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.768666 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.768669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.769108 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.769110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.769273 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.769848 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:02.769858 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:02.771538 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:02.773043 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:02.773049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:02.773051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:02.773271 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:02.773423 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:02.773424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:02.773424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:02.773516 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:02.773802 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:02.773807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:02.773920 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 13:09:02.774001 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:09:02.774135 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260926 13:09:02.775190 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46015
I20260926 13:09:02.775315 1991 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:02.775511 1991 heartbeater.cc:511] Master 127.0.98.62:46015 requested a full tablet report, sending...
I20260926 13:09:02.776866 1508 ts_manager.cc:194] Registered new tserver with Master: 1e7a4194970f477cab7e3a244e3e888a (127.0.98.3:42343)
I20260926 13:09:02.777348 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40309
I20260926 13:09:02.779057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 1860
I20260926 13:09:02.779138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:02.779407 392 external_mini_cluster.cc:1428] Setting key 68eca25cb9f77fa42efefde330ed71f3
I20260926 13:09:02.780944 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:09:02.785138 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.776984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41861 (local address 127.0.98.62:46015)
0926 13:09:02.777045 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.777047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.777186 (+ 139us) server_negotiation.cc:407] Connection header received
0926 13:09:02.777320 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.777321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.777331 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.777358 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:02.777894 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.778610 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.779625 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.779915 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.779970 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.782631 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:02.782633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:02.782633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:02.782642 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:02.784059 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.784464 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.784465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.784465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.784520 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.784702 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.784703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.784704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.784810 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:02.784861 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.785010 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.785070 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:09:02.785220 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.776924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46015 (local address 127.0.98.3:41861)
0926 13:09:02.777104 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:02.777137 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:02.777236 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:02.777441 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:02.777441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:02.777720 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:02.777785 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.777788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.778726 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.778727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.779527 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.779529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.779986 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.779987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.780067 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.780500 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:02.780506 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:02.780874 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:02.784138 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:02.784139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:02.784140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:02.784356 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:02.784576 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:02.784577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:02.784577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:02.784644 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:02.784866 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:02.784867 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:02.784915 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 13:09:02.785001 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 13:09:02.785110 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
Sep 26 13:09:02 dist-test-slave-2dw5 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:09:02 dist-test-slave-2dw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:02.804386 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.797351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53994 (local address 127.0.98.62:46015)
0926 13:09:02.797424 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.797425 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.797435 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:02.797458 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.797458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.797468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.797522 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:02.798049 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.798642 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.799449 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.799646 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.799700 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.801969 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:02.801970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:02.801971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:02.801979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:02.803406 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.803704 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.803704 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.803705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.803760 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:02.803903 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:02.803903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:02.803903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:02.803999 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:02.804061 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.804171 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.804226 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:09:02.806315 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:02.808002 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:02.814555 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.810402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42343 (local address 127.0.0.1:37258)
0926 13:09:02.810680 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:02.810690 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:02.810737 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:02.811049 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:02.811052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:02.811054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:02.811120 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.811122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.812552 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.812553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.813595 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.813597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.814243 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.814244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.814307 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.814311 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:02.814394 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 13:09:02.814450 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":96,"threads_started":1}
I20260926 13:09:02.814621 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.810663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.0.98.1:39615)
0926 13:09:02.810816 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.810821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.810957 (+ 136us) server_negotiation.cc:407] Connection header received
0926 13:09:02.811002 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.811004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.811017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.811083 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:02.811234 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.812334 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.813302 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.813999 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.814061 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.814137 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.814497 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.814533 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260926 13:09:02.814555 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.810400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34795 (local address 127.0.0.1:56380)
0926 13:09:02.810567 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:02.810607 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:02.810675 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:02.811061 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:02.811062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:02.811065 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:02.811137 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.811140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.812204 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.812205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.813530 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.813533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.814348 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.814349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.814405 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.814409 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:02.814454 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 13:09:02.814487 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":10,"thread_start_us":42,"threads_started":1}
I20260926 13:09:02.814905 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.810689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.0.98.3:42343)
0926 13:09:02.810890 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.810894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.810904 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:02.810942 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.810944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.810959 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.811035 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:02.811181 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.812477 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.813660 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.814149 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.814212 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.814290 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.814751 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.814792 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260926 13:09:02.814929 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.810570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39615 (local address 127.0.0.1:36598)
0926 13:09:02.810894 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:02.810902 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:02.810949 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:02.811145 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:02.811145 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:02.811147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:02.811175 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.811176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.812423 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.812424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.813212 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.813214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.814170 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.814171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.814248 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.814262 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 13:09:02.814331 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:09:02.814421 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":56,"threads_started":1}
I20260926 13:09:02.814599 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.810518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.0.98.2:34795)
0926 13:09:02.810894 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.810897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.810906 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:02.810937 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.810939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.810955 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.811329 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:02.811388 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.812125 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.813610 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.814291 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.814347 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.814413 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.814468 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.814504 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":82,"threads_started":1}
I20260926 13:09:02.815667 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 99376f7183c044e0baefc8892ad79ef3 (DEFAULT_TABLE table=test-table [id=66d103343f2e4274b4cc54d68dd54e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:02.815671 1638 tablet_service.cc:1467] Processing CreateTablet for tablet 99376f7183c044e0baefc8892ad79ef3 (DEFAULT_TABLE table=test-table [id=66d103343f2e4274b4cc54d68dd54e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:02.815789 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 99376f7183c044e0baefc8892ad79ef3 (DEFAULT_TABLE table=test-table [id=66d103343f2e4274b4cc54d68dd54e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:02.815959 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99376f7183c044e0baefc8892ad79ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:02.816017 1638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99376f7183c044e0baefc8892ad79ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:02.816039 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99376f7183c044e0baefc8892ad79ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:02.818599 2014 tablet_bootstrap.cc:492] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a: Bootstrap starting.
I20260926 13:09:02.819260 2014 tablet_bootstrap.cc:654] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:02.819538 2014 log.cc:824] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:02.819661 2015 tablet_bootstrap.cc:492] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83: Bootstrap starting.
I20260926 13:09:02.819996 2016 tablet_bootstrap.cc:492] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988: Bootstrap starting.
I20260926 13:09:02.820282 2015 tablet_bootstrap.cc:654] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:02.820554 2016 tablet_bootstrap.cc:654] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:02.820607 2015 log.cc:824] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:02.820866 2016 log.cc:824] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:02.821358 2014 tablet_bootstrap.cc:492] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a: No bootstrap required, opened a new log
I20260926 13:09:02.821358 2015 tablet_bootstrap.cc:492] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83: No bootstrap required, opened a new log
I20260926 13:09:02.821426 2015 ts_tablet_manager.cc:1397] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:02.821426 2014 ts_tablet_manager.cc:1397] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 13:09:02.823378 2016 tablet_bootstrap.cc:492] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988: No bootstrap required, opened a new log
I20260926 13:09:02.823442 2016 ts_tablet_manager.cc:1397] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 13:09:02.823417 2014 raft_consensus.cc:348] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } }
I20260926 13:09:02.823405 2015 raft_consensus.cc:348] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } }
I20260926 13:09:02.823580 2015 raft_consensus.cc:374] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:02.823580 2014 raft_consensus.cc:374] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:02.823621 2014 raft_consensus.cc:729] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7a4194970f477cab7e3a244e3e888a, State: Initialized, Role: FOLLOWER
I20260926 13:09:02.823621 2015 raft_consensus.cc:729] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3096ff9c453e4485b2d15b82b9e9bb83, State: Initialized, Role: FOLLOWER
I20260926 13:09:02.823768 2014 consensus_queue.cc:260] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } }
I20260926 13:09:02.823768 2015 consensus_queue.cc:260] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } }
I20260926 13:09:02.824036 2014 ts_tablet_manager.cc:1428] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 13:09:02.824038 2015 ts_tablet_manager.cc:1428] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:09:02.824165 1991 heartbeater.cc:503] Master 127.0.98.62:46015 was elected leader, sending a full tablet report...
I20260926 13:09:02.824219 1851 heartbeater.cc:503] Master 127.0.98.62:46015 was elected leader, sending a full tablet report...
I20260926 13:09:02.825268 2016 raft_consensus.cc:348] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } }
I20260926 13:09:02.825441 2016 raft_consensus.cc:374] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:02.825486 2016 raft_consensus.cc:729] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3a2365c22d4f1fac0029034a354988, State: Initialized, Role: FOLLOWER
I20260926 13:09:02.825603 2016 consensus_queue.cc:260] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } }
I20260926 13:09:02.825861 2016 ts_tablet_manager.cc:1428] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 13:09:02.825875 1711 heartbeater.cc:503] Master 127.0.98.62:46015 was elected leader, sending a full tablet report...
W20260926 13:09:02.853299 1852 tablet.cc:2369] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:09:02.897291 2021 raft_consensus.cc:482] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:02.897423 2021 raft_consensus.cc:504] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } }
I20260926 13:09:02.897806 2021 leader_election.cc:288] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3096ff9c453e4485b2d15b82b9e9bb83 (127.0.98.2:34795), ec3a2365c22d4f1fac0029034a354988 (127.0.98.1:39615)
I20260926 13:09:02.901808 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.898109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39615 (local address 127.0.98.3:33967)
0926 13:09:02.898353 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:02.898363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:02.898418 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:02.898533 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:02.898534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:02.898535 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:02.898589 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.898591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.899484 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.899486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.900739 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.900742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.901514 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.901515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.901598 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.901603 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:09:02.901663 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 13:09:02.901704 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":63,"thread_start_us":101,"threads_started":1}
I20260926 13:09:02.901839 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.898107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43247 (local address 127.0.98.2:34795)
0926 13:09:02.898153 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.898155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.898165 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:02.898390 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.898391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.898403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.898466 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:02.898620 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.899392 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.900848 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.901501 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.901558 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.901567 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.901726 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.901770 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 13:09:02.901865 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.898209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33967 (local address 127.0.98.1:39615)
0926 13:09:02.898252 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.898255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.898414 (+ 159us) server_negotiation.cc:407] Connection header received
0926 13:09:02.898461 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.898462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.898474 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.898525 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:02.898646 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.899392 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.900849 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.901424 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.901503 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.901515 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.901726 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.901809 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:09:02.901902 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.898005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34795 (local address 127.0.98.3:43247)
0926 13:09:02.898088 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:02.898103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:02.898172 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:02.898482 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:02.898483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:02.898484 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:02.898553 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.898555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.899484 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.899486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.900739 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:02.900742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.901566 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.901567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:02.901624 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.901627 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:02.901674 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 13:09:02.901707 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":2}
I20260926 13:09:02.902297 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99376f7183c044e0baefc8892ad79ef3" candidate_uuid: "1e7a4194970f477cab7e3a244e3e888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" is_pre_election: true
I20260926 13:09:02.902334 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99376f7183c044e0baefc8892ad79ef3" candidate_uuid: "1e7a4194970f477cab7e3a244e3e888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3a2365c22d4f1fac0029034a354988" is_pre_election: true
I20260926 13:09:02.902441 1805 raft_consensus.cc:2454] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e7a4194970f477cab7e3a244e3e888a in term 0.
I20260926 13:09:02.902485 1665 raft_consensus.cc:2454] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e7a4194970f477cab7e3a244e3e888a in term 0.
I20260926 13:09:02.902680 1879 leader_election.cc:302] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7a4194970f477cab7e3a244e3e888a, 3096ff9c453e4485b2d15b82b9e9bb83; no voters:
I20260926 13:09:02.902817 2021 raft_consensus.cc:2781] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:02.902909 2021 raft_consensus.cc:482] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:02.902926 2021 raft_consensus.cc:3037] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:02.903894 2021 raft_consensus.cc:504] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } }
I20260926 13:09:02.904014 2021 leader_election.cc:288] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [CANDIDATE]: Term 1 election: Requested vote from peers 3096ff9c453e4485b2d15b82b9e9bb83 (127.0.98.2:34795), ec3a2365c22d4f1fac0029034a354988 (127.0.98.1:39615)
I20260926 13:09:02.904189 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99376f7183c044e0baefc8892ad79ef3" candidate_uuid: "1e7a4194970f477cab7e3a244e3e888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3096ff9c453e4485b2d15b82b9e9bb83"
I20260926 13:09:02.904273 1805 raft_consensus.cc:3037] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:02.904243 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99376f7183c044e0baefc8892ad79ef3" candidate_uuid: "1e7a4194970f477cab7e3a244e3e888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3a2365c22d4f1fac0029034a354988"
I20260926 13:09:02.904433 1665 raft_consensus.cc:3037] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:02.905006 1805 raft_consensus.cc:2454] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e7a4194970f477cab7e3a244e3e888a in term 1.
I20260926 13:09:02.905114 1665 raft_consensus.cc:2454] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e7a4194970f477cab7e3a244e3e888a in term 1.
I20260926 13:09:02.905144 1879 leader_election.cc:302] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7a4194970f477cab7e3a244e3e888a, 3096ff9c453e4485b2d15b82b9e9bb83; no voters:
I20260926 13:09:02.905220 2021 raft_consensus.cc:2781] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:02.905385 2021 raft_consensus.cc:686] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 1 LEADER]: Becoming Leader. State: Replica: 1e7a4194970f477cab7e3a244e3e888a, State: Running, Role: LEADER
I20260926 13:09:02.905475 2021 consensus_queue.cc:237] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } }
I20260926 13:09:02.906308 1508 catalog_manager.cc:5495] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e7a4194970f477cab7e3a244e3e888a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1e7a4194970f477cab7e3a244e3e888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } health_report { overall_health: UNKNOWN } } }
I20260926 13:09:02.935489 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:02.932981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.0.98.3:42343)
0926 13:09:02.933027 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:02.933029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:02.933037 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:02.933114 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:02.933115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:02.933133 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:02.933181 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:02.933298 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.933759 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.934666 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:02.934890 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:02.934949 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:02.935157 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:02.935252 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:02.935305 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:02.935407 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 13:09:02.935426 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:09:02.937263 1665 raft_consensus.cc:1261] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988 [term 1 FOLLOWER]: Refusing update from remote peer 1e7a4194970f477cab7e3a244e3e888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:02.937254 1805 raft_consensus.cc:1261] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 1 FOLLOWER]: Refusing update from remote peer 1e7a4194970f477cab7e3a244e3e888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:02.937573 2021 consensus_queue.cc:1035] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:02.937711 2024 consensus_queue.cc:1035] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:02.940598 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333593673468219392 to 7333593673341317120. Current Snapshot: MvccSnapshot[applied={T|T < 7333593673468219392}]
I20260926 13:09:02.940662 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333593673468219392 to 7333593673341317120. Current Snapshot: MvccSnapshot[applied={T|T < 7333593673468219392}]
I20260926 13:09:02.940717 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333593673468219392 to 7333593673341317120. Current Snapshot: MvccSnapshot[applied={T|T < 7333593673468219392 or (T in {7333593673468219392})}]
I20260926 13:09:02.943984 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:02.944072 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:02.944948 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d: Sending DeleteTablet for 3 replicas of tablet 99376f7183c044e0baefc8892ad79ef3
I20260926 13:09:02.945263 1638 tablet_service.cc:1514] Processing DeleteTablet for tablet 99376f7183c044e0baefc8892ad79ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36598
I20260926 13:09:02.945226 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 99376f7183c044e0baefc8892ad79ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37258
I20260926 13:09:02.945478 2037 tablet_replica.cc:330] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988: stopping tablet replica
I20260926 13:09:02.945464 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 99376f7183c044e0baefc8892ad79ef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56380
I20260926 13:09:02.945578 2037 raft_consensus.cc:2229] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:02.945678 2037 raft_consensus.cc:2258] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:02.945852 2039 tablet_replica.cc:330] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83: stopping tablet replica
I20260926 13:09:02.945947 2039 raft_consensus.cc:2229] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:02.946054 2039 raft_consensus.cc:2258] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:02.946010 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:02.946148 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:02.946123 2037 ts_tablet_manager.cc:1905] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:02.946462 2039 ts_tablet_manager.cc:1905] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:02.947664 1638 tablet_service.cc:1467] Processing CreateTablet for tablet 5810c1bed88e487d88135883725800ec (DEFAULT_TABLE table=test-table [id=52ef606842454b0c8cb38f0f196904dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:02.947691 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 5810c1bed88e487d88135883725800ec (DEFAULT_TABLE table=test-table [id=52ef606842454b0c8cb38f0f196904dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:02.947770 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5810c1bed88e487d88135883725800ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:02.947779 1638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5810c1bed88e487d88135883725800ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:02.947984 2039 ts_tablet_manager.cc:1918] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:02.948047 2039 log.cc:1196] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/99376f7183c044e0baefc8892ad79ef3
I20260926 13:09:02.948278 2039 ts_tablet_manager.cc:1939] T 99376f7183c044e0baefc8892ad79ef3 P 3096ff9c453e4485b2d15b82b9e9bb83: Deleting consensus metadata
I20260926 13:09:02.948802 1495 catalog_manager.cc:4841] TS 3096ff9c453e4485b2d15b82b9e9bb83 (127.0.98.2:34795): tablet 99376f7183c044e0baefc8892ad79ef3 (table test-table [id=66d103343f2e4274b4cc54d68dd54e30]) successfully deleted
I20260926 13:09:02.949033 2015 tablet_bootstrap.cc:492] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83: Bootstrap starting.
I20260926 13:09:02.949410 2015 tablet_bootstrap.cc:654] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:02.949565 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 5810c1bed88e487d88135883725800ec (DEFAULT_TABLE table=test-table [id=52ef606842454b0c8cb38f0f196904dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:02.949661 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5810c1bed88e487d88135883725800ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:02.949695 2016 tablet_bootstrap.cc:492] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988: Bootstrap starting.
I20260926 13:09:02.949957 2015 tablet_bootstrap.cc:492] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83: No bootstrap required, opened a new log
I20260926 13:09:02.949998 2015 ts_tablet_manager.cc:1397] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:02.950003 2016 tablet_bootstrap.cc:654] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:02.950135 2015 raft_consensus.cc:348] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:02.950198 2015 raft_consensus.cc:374] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:02.950217 2015 raft_consensus.cc:729] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3096ff9c453e4485b2d15b82b9e9bb83, State: Initialized, Role: FOLLOWER
I20260926 13:09:02.950261 2015 consensus_queue.cc:260] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:02.950352 2015 ts_tablet_manager.cc:1428] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:02.950945 2014 tablet_bootstrap.cc:492] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a: Bootstrap starting.
I20260926 13:09:02.951360 2014 tablet_bootstrap.cc:654] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:02.951423 2037 ts_tablet_manager.cc:1918] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:02.951496 2037 log.cc:1196] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/wal/wals/99376f7183c044e0baefc8892ad79ef3
I20260926 13:09:02.951694 2016 tablet_bootstrap.cc:492] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988: No bootstrap required, opened a new log
I20260926 13:09:02.951730 2037 ts_tablet_manager.cc:1939] T 99376f7183c044e0baefc8892ad79ef3 P ec3a2365c22d4f1fac0029034a354988: Deleting consensus metadata
I20260926 13:09:02.951735 2016 ts_tablet_manager.cc:1397] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:02.951923 2038 tablet_replica.cc:330] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a: stopping tablet replica
I20260926 13:09:02.951956 2016 raft_consensus.cc:348] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:02.952029 2038 raft_consensus.cc:2229] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:09:02.952040 2016 raft_consensus.cc:374] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:02.952061 2016 raft_consensus.cc:729] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3a2365c22d4f1fac0029034a354988, State: Initialized, Role: FOLLOWER
I20260926 13:09:02.952103 2016 consensus_queue.cc:260] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:02.952165 2038 raft_consensus.cc:2258] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:02.952185 2016 ts_tablet_manager.cc:1428] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:02.952560 2038 ts_tablet_manager.cc:1905] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:02.952689 1495 catalog_manager.cc:4841] TS ec3a2365c22d4f1fac0029034a354988 (127.0.98.1:39615): tablet 99376f7183c044e0baefc8892ad79ef3 (table test-table [id=66d103343f2e4274b4cc54d68dd54e30]) successfully deleted
I20260926 13:09:02.953891 2038 ts_tablet_manager.cc:1918] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:02.953950 2038 log.cc:1196] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/99376f7183c044e0baefc8892ad79ef3
I20260926 13:09:02.954151 2038 ts_tablet_manager.cc:1939] T 99376f7183c044e0baefc8892ad79ef3 P 1e7a4194970f477cab7e3a244e3e888a: Deleting consensus metadata
I20260926 13:09:02.954517 1492 catalog_manager.cc:4841] TS 1e7a4194970f477cab7e3a244e3e888a (127.0.98.3:42343): tablet 99376f7183c044e0baefc8892ad79ef3 (table test-table [id=66d103343f2e4274b4cc54d68dd54e30]) successfully deleted
I20260926 13:09:02.955188 2014 tablet_bootstrap.cc:492] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a: No bootstrap required, opened a new log
I20260926 13:09:02.955240 2014 ts_tablet_manager.cc:1397] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260926 13:09:02.955384 2014 raft_consensus.cc:348] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:02.955454 2014 raft_consensus.cc:374] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:02.955473 2014 raft_consensus.cc:729] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7a4194970f477cab7e3a244e3e888a, State: Initialized, Role: FOLLOWER
I20260926 13:09:02.955518 2014 consensus_queue.cc:260] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:02.955607 2014 ts_tablet_manager.cc:1428] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:03.004595 2022 raft_consensus.cc:482] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:03.004698 2022 raft_consensus.cc:504] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:03.005007 2022 leader_election.cc:288] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3096ff9c453e4485b2d15b82b9e9bb83 (127.0.98.2:34795), 1e7a4194970f477cab7e3a244e3e888a (127.0.98.3:42343)
I20260926 13:09:03.008541 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.005417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49811 (local address 127.0.98.2:34795)
0926 13:09:03.005458 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:03.005459 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:03.005531 (+ 72us) server_negotiation.cc:407] Connection header received
0926 13:09:03.005590 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:03.005591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:03.005599 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:03.005645 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:03.005784 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.006533 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.007849 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.008195 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.008254 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.008265 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:03.008435 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 13:09:03.008470 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:09:03.008560 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.005169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34795 (local address 127.0.98.1:49811)
0926 13:09:03.005454 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:03.005466 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:03.005535 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:03.005655 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:03.005655 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:03.005657 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:03.005723 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:03.005725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.006608 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.006609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:03.007732 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:03.007734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.008290 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.008291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:03.008365 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.008368 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:03.008421 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 13:09:03.008462 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":4,"thread_start_us":59,"threads_started":1}
I20260926 13:09:03.008775 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5810c1bed88e487d88135883725800ec" candidate_uuid: "ec3a2365c22d4f1fac0029034a354988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" is_pre_election: true
I20260926 13:09:03.008869 1805 raft_consensus.cc:2454] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3a2365c22d4f1fac0029034a354988 in term 0.
I20260926 13:09:03.008934 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.005282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40527 (local address 127.0.98.3:42343)
0926 13:09:03.005436 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:03.005438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:03.005557 (+ 119us) server_negotiation.cc:407] Connection header received
0926 13:09:03.005639 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:03.005640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:03.005649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:03.005694 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:03.005844 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.006357 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.007825 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.008353 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.008431 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.008442 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:03.008838 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 13:09:03.008878 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260926 13:09:03.009027 1599 leader_election.cc:302] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3096ff9c453e4485b2d15b82b9e9bb83, ec3a2365c22d4f1fac0029034a354988; no voters:
I20260926 13:09:03.009143 2022 raft_consensus.cc:2781] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:03.009155 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.005173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42343 (local address 127.0.98.1:40527)
0926 13:09:03.005493 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:03.005503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:03.005583 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:03.005698 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:03.005699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:03.005700 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:03.005764 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:03.005765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.006432 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.006434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:03.007731 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:03.007734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.008678 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.008679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:03.008745 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.008749 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:03.009045 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 13:09:03.009085 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":34,"thread_start_us":61,"threads_started":1}
I20260926 13:09:03.009194 2022 raft_consensus.cc:482] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:03.009243 2022 raft_consensus.cc:3037] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:03.009496 1928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5810c1bed88e487d88135883725800ec" candidate_uuid: "ec3a2365c22d4f1fac0029034a354988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7a4194970f477cab7e3a244e3e888a" is_pre_election: true
I20260926 13:09:03.009618 1928 raft_consensus.cc:2454] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3a2365c22d4f1fac0029034a354988 in term 0.
I20260926 13:09:03.010066 2022 raft_consensus.cc:504] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:03.010185 2022 leader_election.cc:288] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [CANDIDATE]: Term 1 election: Requested vote from peers 3096ff9c453e4485b2d15b82b9e9bb83 (127.0.98.2:34795), 1e7a4194970f477cab7e3a244e3e888a (127.0.98.3:42343)
I20260926 13:09:03.010443 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5810c1bed88e487d88135883725800ec" candidate_uuid: "ec3a2365c22d4f1fac0029034a354988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3096ff9c453e4485b2d15b82b9e9bb83"
I20260926 13:09:03.010437 1928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5810c1bed88e487d88135883725800ec" candidate_uuid: "ec3a2365c22d4f1fac0029034a354988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e7a4194970f477cab7e3a244e3e888a"
I20260926 13:09:03.010510 1805 raft_consensus.cc:3037] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:03.010509 1928 raft_consensus.cc:3037] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:03.011142 1805 raft_consensus.cc:2454] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3a2365c22d4f1fac0029034a354988 in term 1.
I20260926 13:09:03.011298 1599 leader_election.cc:302] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3096ff9c453e4485b2d15b82b9e9bb83, ec3a2365c22d4f1fac0029034a354988; no voters:
I20260926 13:09:03.011371 2022 raft_consensus.cc:2781] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:03.011446 1928 raft_consensus.cc:2454] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3a2365c22d4f1fac0029034a354988 in term 1.
I20260926 13:09:03.011472 2022 raft_consensus.cc:686] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 1 LEADER]: Becoming Leader. State: Replica: ec3a2365c22d4f1fac0029034a354988, State: Running, Role: LEADER
I20260926 13:09:03.011540 2022 consensus_queue.cc:237] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:03.012331 1507 catalog_manager.cc:5495] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec3a2365c22d4f1fac0029034a354988 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ec3a2365c22d4f1fac0029034a354988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } health_report { overall_health: UNKNOWN } } }
W20260926 13:09:03.021790 1992 tablet.cc:2369] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 13:09:03 dist-test-slave-2dw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:03.047191 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.039614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.0.98.62:46015)
0926 13:09:03.039675 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:03.039677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:03.039816 (+ 139us) server_negotiation.cc:407] Connection header received
0926 13:09:03.039848 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:03.039849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:03.039859 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:03.039914 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:03.040482 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.040971 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.041937 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.042141 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.042290 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.044597 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:03.044598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:03.044599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:03.044605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:03.046053 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:03.046473 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:03.046474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:03.046475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:03.046524 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:03.046708 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:03.046708 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:03.046708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:03.046806 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:03.046886 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:03.047048 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 13:09:03.047123 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:09:03.047302 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.039547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46015 (local address 127.0.0.1:54002)
0926 13:09:03.039710 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:03.039722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:03.039803 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:03.039944 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:03.039945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:03.040347 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:03.040404 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:03.040407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.041048 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.041049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:03.041830 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:03.041833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.042223 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.042224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:03.042283 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.042896 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:03.042902 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:03.044504 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:03.046139 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:03.046147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:03.046149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:03.046390 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:03.046580 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:03.046581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:03.046581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:03.046652 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:03.046880 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:03.046888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:03.046952 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 13:09:03.047028 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 13:09:03.047196 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":42,"thread_start_us":65,"threads_started":1}
I20260926 13:09:03.049510 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54002:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 13:09:03.049719 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:03.051097 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 97fc87f6e7984c5bbaab440886572093 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=049d1c0a388d44739655a80f69d94e0b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:09:03.051174 1638 tablet_service.cc:1467] Processing CreateTablet for tablet 97fc87f6e7984c5bbaab440886572093 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=049d1c0a388d44739655a80f69d94e0b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:09:03.051213 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 97fc87f6e7984c5bbaab440886572093 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=049d1c0a388d44739655a80f69d94e0b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:09:03.051270 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97fc87f6e7984c5bbaab440886572093. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:03.051285 1638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97fc87f6e7984c5bbaab440886572093. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:03.051349 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97fc87f6e7984c5bbaab440886572093. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:03.052778 2014 tablet_bootstrap.cc:492] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: Bootstrap starting.
I20260926 13:09:03.053180 2014 tablet_bootstrap.cc:654] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:03.053508 2016 tablet_bootstrap.cc:492] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988: Bootstrap starting.
I20260926 13:09:03.053867 2016 tablet_bootstrap.cc:654] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:03.054208 2015 tablet_bootstrap.cc:492] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83: Bootstrap starting.
I20260926 13:09:03.054589 2015 tablet_bootstrap.cc:654] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:03.055104 2016 tablet_bootstrap.cc:492] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988: No bootstrap required, opened a new log
I20260926 13:09:03.055151 2016 ts_tablet_manager.cc:1397] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:03.055408 2016 raft_consensus.cc:348] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:03.055495 2016 raft_consensus.cc:374] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:03.055516 2016 raft_consensus.cc:729] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3a2365c22d4f1fac0029034a354988, State: Initialized, Role: FOLLOWER
I20260926 13:09:03.055555 2016 consensus_queue.cc:260] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:03.055645 2016 ts_tablet_manager.cc:1428] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:03.055759 2015 tablet_bootstrap.cc:492] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83: No bootstrap required, opened a new log
I20260926 13:09:03.055774 2022 tablet_replica.cc:439] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260926 13:09:03.055810 2015 ts_tablet_manager.cc:1397] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 13:09:03.055840 2022 tablet_replica.cc:442] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:03.055959 2015 raft_consensus.cc:348] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:03.056039 2015 raft_consensus.cc:374] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:03.056063 2015 raft_consensus.cc:729] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3096ff9c453e4485b2d15b82b9e9bb83, State: Initialized, Role: FOLLOWER
I20260926 13:09:03.056059 2014 tablet_bootstrap.cc:492] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: No bootstrap required, opened a new log
I20260926 13:09:03.056102 2014 ts_tablet_manager.cc:1397] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 13:09:03.056100 2015 consensus_queue.cc:260] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:03.056170 2015 ts_tablet_manager.cc:1428] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:03.056255 2014 raft_consensus.cc:348] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:03.056330 2014 raft_consensus.cc:374] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:03.056348 2014 raft_consensus.cc:729] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e7a4194970f477cab7e3a244e3e888a, State: Initialized, Role: FOLLOWER
I20260926 13:09:03.056386 2014 consensus_queue.cc:260] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:03.056399 2020 tablet_replica.cc:439] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260926 13:09:03.056465 2014 ts_tablet_manager.cc:1428] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:03.056479 2020 tablet_replica.cc:442] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:03.056561 2032 tablet_replica.cc:439] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260926 13:09:03.056638 2032 tablet_replica.cc:442] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:03.176635 2032 raft_consensus.cc:482] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:03.176745 2032 raft_consensus.cc:504] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:03.176908 2032 leader_election.cc:288] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec3a2365c22d4f1fac0029034a354988 (127.0.98.1:39615), 3096ff9c453e4485b2d15b82b9e9bb83 (127.0.98.2:34795)
I20260926 13:09:03.177136 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fc87f6e7984c5bbaab440886572093" candidate_uuid: "1e7a4194970f477cab7e3a244e3e888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3a2365c22d4f1fac0029034a354988" is_pre_election: true
I20260926 13:09:03.177160 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fc87f6e7984c5bbaab440886572093" candidate_uuid: "1e7a4194970f477cab7e3a244e3e888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" is_pre_election: true
I20260926 13:09:03.177246 1805 raft_consensus.cc:2454] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e7a4194970f477cab7e3a244e3e888a in term 0.
I20260926 13:09:03.177246 1665 raft_consensus.cc:2454] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e7a4194970f477cab7e3a244e3e888a in term 0.
I20260926 13:09:03.177398 1879 leader_election.cc:302] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7a4194970f477cab7e3a244e3e888a, 3096ff9c453e4485b2d15b82b9e9bb83; no voters:
I20260926 13:09:03.177516 2032 raft_consensus.cc:2781] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:03.177560 2032 raft_consensus.cc:482] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:03.177591 2032 raft_consensus.cc:3037] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:03.178419 2032 raft_consensus.cc:504] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:03.178541 2032 leader_election.cc:288] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [CANDIDATE]: Term 1 election: Requested vote from peers ec3a2365c22d4f1fac0029034a354988 (127.0.98.1:39615), 3096ff9c453e4485b2d15b82b9e9bb83 (127.0.98.2:34795)
I20260926 13:09:03.178743 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fc87f6e7984c5bbaab440886572093" candidate_uuid: "1e7a4194970f477cab7e3a244e3e888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3a2365c22d4f1fac0029034a354988"
I20260926 13:09:03.178767 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97fc87f6e7984c5bbaab440886572093" candidate_uuid: "1e7a4194970f477cab7e3a244e3e888a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3096ff9c453e4485b2d15b82b9e9bb83"
I20260926 13:09:03.178826 1665 raft_consensus.cc:3037] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:03.178869 1805 raft_consensus.cc:3037] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:03.179520 1665 raft_consensus.cc:2454] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e7a4194970f477cab7e3a244e3e888a in term 1.
I20260926 13:09:03.179531 1805 raft_consensus.cc:2454] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e7a4194970f477cab7e3a244e3e888a in term 1.
I20260926 13:09:03.179689 1879 leader_election.cc:302] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e7a4194970f477cab7e3a244e3e888a, ec3a2365c22d4f1fac0029034a354988; no voters:
I20260926 13:09:03.179831 2032 raft_consensus.cc:2781] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:03.179896 2032 raft_consensus.cc:686] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [term 1 LEADER]: Becoming Leader. State: Replica: 1e7a4194970f477cab7e3a244e3e888a, State: Running, Role: LEADER
I20260926 13:09:03.179948 2032 consensus_queue.cc:237] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } }
I20260926 13:09:03.180222 2021 tablet_replica.cc:439] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: TxnStatusTablet state changed. Reason: New leader 1e7a4194970f477cab7e3a244e3e888a. Latest consensus state: current_term: 1 leader_uuid: "1e7a4194970f477cab7e3a244e3e888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260926 13:09:03.180305 2021 tablet_replica.cc:442] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:09:03.180435 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 13:09:03.180538 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 13:09:03.180570 1507 catalog_manager.cc:5495] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e7a4194970f477cab7e3a244e3e888a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1e7a4194970f477cab7e3a244e3e888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } health_report { overall_health: HEALTHY } } }
W20260926 13:09:03.190616 1712 tablet.cc:2369] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:09:03.203672 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.200941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42343 (local address 127.0.0.1:37272)
0926 13:09:03.201020 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:03.201032 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:03.201078 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:03.201224 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:03.201224 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:03.201225 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:03.201272 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:03.201274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.202123 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.202124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:03.202784 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:03.202788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.203304 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.203305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:03.203368 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.203383 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:03.203542 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:03.203546 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:03.203575 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 13:09:03.203612 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 13:09:03.203691 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.201001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.0.98.3:42343)
0926 13:09:03.201055 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:03.201057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:03.201069 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:09:03.201151 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:03.201152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:03.201165 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:03.201223 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:03.201326 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.202047 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.202912 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.203226 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.203295 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.203456 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:03.203490 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:03.203545 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:03.203582 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 13:09:03.203604 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:09:03.204730 1805 raft_consensus.cc:1261] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83 [term 1 FOLLOWER]: Refusing update from remote peer 1e7a4194970f477cab7e3a244e3e888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:03.204767 1665 raft_consensus.cc:1261] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988 [term 1 FOLLOWER]: Refusing update from remote peer 1e7a4194970f477cab7e3a244e3e888a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:03.204981 2021 consensus_queue.cc:1035] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:03.205102 2032 consensus_queue.cc:1035] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:03.205533 2022 tablet_replica.cc:439] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988: TxnStatusTablet state changed. Reason: New leader 1e7a4194970f477cab7e3a244e3e888a. Latest consensus state: current_term: 1 leader_uuid: "1e7a4194970f477cab7e3a244e3e888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260926 13:09:03.205631 2022 tablet_replica.cc:442] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:03.206069 2020 tablet_replica.cc:439] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83: TxnStatusTablet state changed. Reason: New leader 1e7a4194970f477cab7e3a244e3e888a. Latest consensus state: current_term: 1 leader_uuid: "1e7a4194970f477cab7e3a244e3e888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260926 13:09:03.206171 2020 tablet_replica.cc:442] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:03.206574 2021 tablet_replica.cc:439] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e7a4194970f477cab7e3a244e3e888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260926 13:09:03.206657 2021 tablet_replica.cc:442] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:09:03.207590 2021 tablet_replica.cc:439] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e7a4194970f477cab7e3a244e3e888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260926 13:09:03.207662 2021 tablet_replica.cc:442] T 97fc87f6e7984c5bbaab440886572093 P 1e7a4194970f477cab7e3a244e3e888a: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:09:03.207657 2020 tablet_replica.cc:439] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e7a4194970f477cab7e3a244e3e888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260926 13:09:03.207757 2020 tablet_replica.cc:442] T 97fc87f6e7984c5bbaab440886572093 P 3096ff9c453e4485b2d15b82b9e9bb83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:03.208344 2022 tablet_replica.cc:439] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e7a4194970f477cab7e3a244e3e888a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3a2365c22d4f1fac0029034a354988" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39615 } } peers { permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 } } peers { permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 } } }
I20260926 13:09:03.208426 2022 tablet_replica.cc:442] T 97fc87f6e7984c5bbaab440886572093 P ec3a2365c22d4f1fac0029034a354988: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:03.211133 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.208527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.0.98.1:39615)
0926 13:09:03.208583 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:03.208585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:03.208593 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:03.208625 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:03.208626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:03.208646 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:03.208714 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:03.208862 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.209391 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.210411 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.210595 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.210648 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.210804 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:03.210938 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:03.210991 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:03.211059 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 13:09:03.211077 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:09:03.216640 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.213605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42167 (local address 127.0.98.3:42343)
0926 13:09:03.213649 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:03.213650 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:03.214168 (+ 518us) server_negotiation.cc:407] Connection header received
0926 13:09:03.214200 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:03.214201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:03.214212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:03.214285 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:03.214378 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.215088 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.215869 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.216147 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.216220 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.216377 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:03.216408 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:03.216463 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:03.216545 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 13:09:03.216567 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:09:03.216639 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.213547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42343 (local address 127.0.98.1:42167)
0926 13:09:03.213670 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:03.213680 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:03.214165 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:03.214289 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:03.214289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:03.214290 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:03.214340 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:03.214342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.215160 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.215161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:03.215796 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:03.215798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.216229 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.216230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:03.216280 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.216298 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:03.216485 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:03.216488 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:03.216531 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 13:09:03.216567 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":68,"threads_started":1}
I20260926 13:09:03.218937 1928 raft_consensus.cc:1261] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a [term 1 FOLLOWER]: Refusing update from remote peer ec3a2365c22d4f1fac0029034a354988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:03.218961 1805 raft_consensus.cc:1261] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83 [term 1 FOLLOWER]: Refusing update from remote peer ec3a2365c22d4f1fac0029034a354988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:03.219199 2045 consensus_queue.cc:1035] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3096ff9c453e4485b2d15b82b9e9bb83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:03.219445 2022 consensus_queue.cc:1035] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e7a4194970f477cab7e3a244e3e888a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:03.228165 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.225305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39615 (local address 127.0.98.3:49087)
0926 13:09:03.225421 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:03.225440 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:03.225487 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:03.225609 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:03.225609 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:03.225610 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:03.225648 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:03.225649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.226517 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.226518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:03.227339 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:03.227341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.227801 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.227802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:03.227860 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.227875 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:03.228045 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:03.228047 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:09:03.228075 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 13:09:03.228109 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 13:09:03.228179 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.225395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49087 (local address 127.0.98.1:39615)
0926 13:09:03.225439 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:03.225441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:03.225476 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:09:03.225526 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:03.225527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:03.225541 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:03.225608 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:03.225685 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.226438 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.227409 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.227696 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.227782 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.227952 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:03.227988 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:03.228046 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:03.228080 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 13:09:03.228100 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:09:03.238190 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:03.238323 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:03.239054 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5679fb23842f4122b4e37738dd80c43d: Sending DeleteTablet for 3 replicas of tablet 5810c1bed88e487d88135883725800ec
I20260926 13:09:03.239428 1638 tablet_service.cc:1514] Processing DeleteTablet for tablet 5810c1bed88e487d88135883725800ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36598
I20260926 13:09:03.239530 2037 tablet_replica.cc:330] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988: stopping tablet replica
I20260926 13:09:03.239580 2037 raft_consensus.cc:2229] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:09:03.239674 2037 raft_consensus.cc:2258] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:03.240092 2037 ts_tablet_manager.cc:1905] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:03.241847 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 5810c1bed88e487d88135883725800ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37258
I20260926 13:09:03.242002 2038 tablet_replica.cc:330] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a: stopping tablet replica
I20260926 13:09:03.242002 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 5810c1bed88e487d88135883725800ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56380
I20260926 13:09:03.242039 2037 ts_tablet_manager.cc:1918] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:03.242066 2038 raft_consensus.cc:2229] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:03.242091 2037 log.cc:1196] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-0/wal/wals/5810c1bed88e487d88135883725800ec
I20260926 13:09:03.242125 2038 raft_consensus.cc:2258] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:03.242142 2039 tablet_replica.cc:330] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83: stopping tablet replica
I20260926 13:09:03.242189 2039 raft_consensus.cc:2229] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:03.242228 2039 raft_consensus.cc:2258] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:03.242305 2037 ts_tablet_manager.cc:1939] T 5810c1bed88e487d88135883725800ec P ec3a2365c22d4f1fac0029034a354988: Deleting consensus metadata
I20260926 13:09:03.242769 2039 ts_tablet_manager.cc:1905] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:03.242774 1495 catalog_manager.cc:4841] TS ec3a2365c22d4f1fac0029034a354988 (127.0.98.1:39615): tablet 5810c1bed88e487d88135883725800ec (table test-table [id=52ef606842454b0c8cb38f0f196904dc]) successfully deleted
I20260926 13:09:03.243067 2038 ts_tablet_manager.cc:1905] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:03.244612 2039 ts_tablet_manager.cc:1918] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:03.244675 2039 log.cc:1196] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/5810c1bed88e487d88135883725800ec
I20260926 13:09:03.244925 2039 ts_tablet_manager.cc:1939] T 5810c1bed88e487d88135883725800ec P 3096ff9c453e4485b2d15b82b9e9bb83: Deleting consensus metadata
I20260926 13:09:03.245266 1495 catalog_manager.cc:4841] TS 3096ff9c453e4485b2d15b82b9e9bb83 (127.0.98.2:34795): tablet 5810c1bed88e487d88135883725800ec (table test-table [id=52ef606842454b0c8cb38f0f196904dc]) successfully deleted
I20260926 13:09:03.245416 2038 ts_tablet_manager.cc:1918] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:03.245479 2038 log.cc:1196] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/5810c1bed88e487d88135883725800ec
I20260926 13:09:03.245693 2038 ts_tablet_manager.cc:1939] T 5810c1bed88e487d88135883725800ec P 1e7a4194970f477cab7e3a244e3e888a: Deleting consensus metadata
I20260926 13:09:03.246014 1492 catalog_manager.cc:4841] TS 1e7a4194970f477cab7e3a244e3e888a (127.0.98.3:42343): tablet 5810c1bed88e487d88135883725800ec (table test-table [id=52ef606842454b0c8cb38f0f196904dc]) successfully deleted
Sep 26 13:09:03 dist-test-slave-2dw5 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428142, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:09:03.256554 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.240656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.0.98.1:39615)
0926 13:09:03.241939 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:03.241941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:03.246275 (+ 4334us) server_negotiation.cc:407] Connection header received
0926 13:09:03.246402 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:03.246403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:03.246414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:03.246444 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:03.247254 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.247970 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.249026 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.249325 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.249369 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.253073 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:03.253083 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:03.253083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:03.253092 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:03.255160 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:03.255586 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:03.255590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:03.255591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:03.255680 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:03.255904 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:03.255905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:03.255905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:03.256093 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:03.256245 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:03.256329 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 13:09:03.256419 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1258}
W20260926 13:09:03.256845 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36616
I20260926 13:09:03.265869 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:03.257882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.0.98.62:46015)
0926 13:09:03.257931 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:03.257933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:03.258156 (+ 223us) server_negotiation.cc:407] Connection header received
0926 13:09:03.258210 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:03.258210 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:03.258223 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:03.258289 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:03.258926 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.259749 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.260858 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:03.261166 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:03.261247 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:03.262770 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:03.262772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:03.262772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:03.262780 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:03.264602 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:03.265037 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:03.265039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:03.265039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:03.265108 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:03.265309 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:03.265310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:03.265311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:03.265456 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:03.265564 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:03.265676 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 13:09:03.265769 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 13:09:03.266212 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54010
I20260926 13:09:03.267086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 1580
I20260926 13:09:03.273484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 1720
I20260926 13:09:03.278199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 1860
I20260926 13:09:03.283299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 1476
2026-09-26T13:09:03Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2238 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:03 dist-test-slave-2dw5 krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:03 dist-test-slave-2dw5 krb5kdc[2077](info): set up 2 sockets
Sep 26 13:09:03 dist-test-slave-2dw5 krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260926 13:09:04.246456 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:04 dist-test-slave-2dw5 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:09:04.366765 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:04.450341 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:04.450543 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:04.450564 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:04.451694 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:04.451731 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 13:09:04.451742 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:04.451753 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:04.451764 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:04.451774 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:04.453239 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44863
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:04.453436 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:04.453609 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:04.458333 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:04.458398 2090 server_base.cc:1034] running on GCE node
W20260926 13:09:04.458413 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:04.458349 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:04.458865 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:04.459151 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:04.460292 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790428144460278 us; error 29 us; skew 500 ppm
Sep 26 13:09:04 dist-test-slave-2dw5 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:04.462795 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:09:04.463260 2090 webserver.cc:460] Webserver started at http://127.0.98.62:44533/ using document root <none> and password file <none>
I20260926 13:09:04.463481 2090 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:04.463531 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:04.463651 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:04.464673 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "8f4d5984c7e741988487dfbc0ead00c7"
format_stamp: "Formatted at 2026-09-26 13:09:04 on dist-test-slave-2dw5"
server_key: "cc2ec9ad07df8ad9c5033fa5e6138526"
server_key_iv: "d5b88635707ea5657ce3031386488e21"
server_key_version: "encryptionkey@0"
I20260926 13:09:04.464984 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f4d5984c7e741988487dfbc0ead00c7"
format_stamp: "Formatted at 2026-09-26 13:09:04 on dist-test-slave-2dw5"
server_key: "cc2ec9ad07df8ad9c5033fa5e6138526"
server_key_iv: "d5b88635707ea5657ce3031386488e21"
server_key_version: "encryptionkey@0"
I20260926 13:09:04.466229 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 13:09:04.466876 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:04.467038 2090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:04.467116 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "8f4d5984c7e741988487dfbc0ead00c7"
format_stamp: "Formatted at 2026-09-26 13:09:04 on dist-test-slave-2dw5"
server_key: "cc2ec9ad07df8ad9c5033fa5e6138526"
server_key_iv: "d5b88635707ea5657ce3031386488e21"
server_key_version: "encryptionkey@0"
I20260926 13:09:04.467181 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:04.476190 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:04.478302 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:04.478443 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:04.483417 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44863
I20260926 13:09:04.483469 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44863 every 8 connection(s)
I20260926 13:09:04.483800 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:04.484493 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:04.486800 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7: Bootstrap starting.
I20260926 13:09:04.487424 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:04.487684 2168 log.cc:824] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:04.488337 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7: No bootstrap required, opened a new log
I20260926 13:09:04.490439 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f4d5984c7e741988487dfbc0ead00c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44863 } }
I20260926 13:09:04.490592 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:04.490613 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4d5984c7e741988487dfbc0ead00c7, State: Initialized, Role: FOLLOWER
I20260926 13:09:04.490741 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f4d5984c7e741988487dfbc0ead00c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44863 } }
I20260926 13:09:04.490814 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:04.490866 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:04.490914 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:04.491432 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f4d5984c7e741988487dfbc0ead00c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44863 } }
I20260926 13:09:04.491513 2168 leader_election.cc:302] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f4d5984c7e741988487dfbc0ead00c7; no voters:
I20260926 13:09:04.491634 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:04.491706 2171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:04.491850 2171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [term 1 LEADER]: Becoming Leader. State: Replica: 8f4d5984c7e741988487dfbc0ead00c7, State: Running, Role: LEADER
I20260926 13:09:04.491936 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:04.491940 2171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f4d5984c7e741988487dfbc0ead00c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44863 } }
I20260926 13:09:04.492264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 2090
I20260926 13:09:04.492349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:04.492339 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f4d5984c7e741988487dfbc0ead00c7. Latest consensus state: current_term: 1 leader_uuid: "8f4d5984c7e741988487dfbc0ead00c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f4d5984c7e741988487dfbc0ead00c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44863 } } }
I20260926 13:09:04.492408 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:04.492535 392 external_mini_cluster.cc:1428] Setting key e604e3872df5a0f3ef29158fcc39af0c
I20260926 13:09:04.492547 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f4d5984c7e741988487dfbc0ead00c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f4d5984c7e741988487dfbc0ead00c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44863 } } }
I20260926 13:09:04.492594 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:04.492995 2177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:04.493537 2177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:04.495184 2177 catalog_manager.cc:1347] Generated new cluster ID: bcd430a429a64ffe892830641493ee68
I20260926 13:09:04.495240 2177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:09:04 dist-test-slave-2dw5 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:04.502144 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:04.492983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.0.98.62:44863)
0926 13:09:04.493327 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:04.493334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:04.493368 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:09:04.493491 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:04.493498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:04.493555 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:04.494269 (+ 714us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:04.494369 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.495625 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.496430 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.496870 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.496985 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:04.499163 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:04.499167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:04.499167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:04.499178 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:04.500702 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:04.501028 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:04.501031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:04.501031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:04.501136 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:04.501308 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:04.501308 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:04.501309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:04.501551 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:04.501693 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:04.501790 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 13:09:04.501928 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":94,"threads_started":1}
I20260926 13:09:04.505877 2177 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:04.506601 2177 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:04.508872 2177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7: Generated new TSK 0
I20260926 13:09:04.509038 2177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:09:04.542259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44863
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:04.660605 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:04.660899 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:04.660939 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:04.662770 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:04.662830 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:04.662896 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 13:09:04.662918 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:09:04.662948 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:04.665205 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:04.665470 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:04.665724 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:04.671062 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:04.671118 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:04.671065 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:04.671314 2194 server_base.cc:1034] running on GCE node
I20260926 13:09:04.671655 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:04.671895 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:04.673062 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790428144673047 us; error 27 us; skew 500 ppm
Sep 26 13:09:04 dist-test-slave-2dw5 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:04.675953 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:09:04.676381 2194 webserver.cc:460] Webserver started at http://127.0.98.1:41873/ using document root <none> and password file <none>
I20260926 13:09:04.676599 2194 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:04.676643 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:04.676749 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:04.677930 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "04abd56465fe4d14b2c1056ea7649d49"
format_stamp: "Formatted at 2026-09-26 13:09:04 on dist-test-slave-2dw5"
server_key: "a8667d5dc55137ebbb0ea1017a54bf79"
server_key_iv: "70ad8e66368bc6bcce3584890667dbe3"
server_key_version: "encryptionkey@0"
I20260926 13:09:04.678263 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "04abd56465fe4d14b2c1056ea7649d49"
format_stamp: "Formatted at 2026-09-26 13:09:04 on dist-test-slave-2dw5"
server_key: "a8667d5dc55137ebbb0ea1017a54bf79"
server_key_iv: "70ad8e66368bc6bcce3584890667dbe3"
server_key_version: "encryptionkey@0"
I20260926 13:09:04.679574 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:09:04.680276 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:04.680490 2194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:09:04.680568 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "04abd56465fe4d14b2c1056ea7649d49"
format_stamp: "Formatted at 2026-09-26 13:09:04 on dist-test-slave-2dw5"
server_key: "a8667d5dc55137ebbb0ea1017a54bf79"
server_key_iv: "70ad8e66368bc6bcce3584890667dbe3"
server_key_version: "encryptionkey@0"
I20260926 13:09:04.680614 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:04.694029 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:04.696309 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:04.696437 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:04.697090 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:04.697129 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:04.697151 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:04.697168 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:04.703702 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40577
I20260926 13:09:04.703830 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40577 every 8 connection(s)
I20260926 13:09:04.704107 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:09:04 dist-test-slave-2dw5 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:04.707569 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:04.697663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34735 (local address 127.0.98.62:44863)
0926 13:09:04.697706 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:04.697708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:04.697986 (+ 278us) server_negotiation.cc:407] Connection header received
0926 13:09:04.698550 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:04.698551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:04.698565 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:04.698632 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:04.699501 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.700278 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.701906 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.702159 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.702215 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:04.704908 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:04.704909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:04.704909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:04.704916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:04.706277 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:04.706703 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:04.706704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:04.706704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:04.706751 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:04.707011 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:04.707011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:04.707011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:04.707152 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:04.707206 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:04.707438 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 13:09:04.707505 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:09:04.707772 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:04.697654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44863 (local address 127.0.98.1:34735)
0926 13:09:04.697828 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:04.697864 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:04.698423 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:04.698731 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:04.698736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:04.699117 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:04.699415 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:04.699421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.700390 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.700391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:04.701807 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:04.701811 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.702251 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.702252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:04.702419 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:04.703090 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:04.703101 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:04.704829 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:04.706369 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:04.706376 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:04.706378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:04.706636 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:04.706794 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:04.706795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:04.706795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:04.706933 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:04.707204 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:04.707208 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:04.707331 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 13:09:04.707411 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 13:09:04.707555 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260926 13:09:04.708187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 2194
I20260926 13:09:04.708263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:04.708335 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44863
I20260926 13:09:04.708422 2325 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:04.708455 392 external_mini_cluster.cc:1428] Setting key 824c5777ef7b1dc191248b2b507e9553
I20260926 13:09:04.708626 2325 heartbeater.cc:511] Master 127.0.98.62:44863 requested a full tablet report, sending...
I20260926 13:09:04.709383 2122 ts_manager.cc:194] Registered new tserver with Master: 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577)
I20260926 13:09:04.710117 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34735
I20260926 13:09:04.714709 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:04.709289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49789 (local address 127.0.98.62:44863)
0926 13:09:04.709345 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:04.709347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:04.709480 (+ 133us) server_negotiation.cc:407] Connection header received
0926 13:09:04.709559 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:04.709560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:04.709570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:04.709626 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:04.709946 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.710636 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.711386 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.711585 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.711649 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:04.712500 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:04.712501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:04.712501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:04.712508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:04.713731 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:04.714028 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:04.714029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:04.714029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:04.714083 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:04.714242 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:04.714243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:04.714243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:04.714352 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:04.714404 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:04.714535 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 13:09:04.714611 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:09:04.714725 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:04.709239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44863 (local address 127.0.98.1:49789)
0926 13:09:04.709400 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:04.709411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:04.709504 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:04.709628 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:04.709629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:04.709814 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:04.709880 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:04.709881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.710742 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.710743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:04.711313 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:04.711315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.711647 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.711647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:04.711715 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:04.712128 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:04.712133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:04.712420 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:04.713806 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:04.713807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:04.713807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:04.713970 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:04.714138 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:04.714138 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:04.714138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:04.714194 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:04.714403 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:04.714403 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:04.714449 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 13:09:04.714508 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:09:04.714612 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:09:04.734571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44863
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:04.816519 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:04.816745 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:04.816772 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:04.818024 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:04.818076 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:04.818097 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 13:09:04.818109 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:09:04.818126 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:04.819754 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:04.819978 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:04.820211 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:04.825271 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:04.825300 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:04.825505 2334 server_base.cc:1034] running on GCE node
W20260926 13:09:04.825628 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:04.825860 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:04.826099 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:04.827253 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790428144827227 us; error 34 us; skew 500 ppm
Sep 26 13:09:04 dist-test-slave-2dw5 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:04.830003 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:09:04.830484 2334 webserver.cc:460] Webserver started at http://127.0.98.2:35543/ using document root <none> and password file <none>
I20260926 13:09:04.830674 2334 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:04.830717 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:04.830816 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:04.831954 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "b4d1a6afd64d42f8802cf1950d495fd9"
format_stamp: "Formatted at 2026-09-26 13:09:04 on dist-test-slave-2dw5"
server_key: "b47950cf7e0c65b29002ac0e1e184963"
server_key_iv: "43212eadd25c26202c1de0d0213f0f20"
server_key_version: "encryptionkey@0"
I20260926 13:09:04.832294 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b4d1a6afd64d42f8802cf1950d495fd9"
format_stamp: "Formatted at 2026-09-26 13:09:04 on dist-test-slave-2dw5"
server_key: "b47950cf7e0c65b29002ac0e1e184963"
server_key_iv: "43212eadd25c26202c1de0d0213f0f20"
server_key_version: "encryptionkey@0"
I20260926 13:09:04.833766 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:04.834605 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:04.834774 2334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:04.834887 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "b4d1a6afd64d42f8802cf1950d495fd9"
format_stamp: "Formatted at 2026-09-26 13:09:04 on dist-test-slave-2dw5"
server_key: "b47950cf7e0c65b29002ac0e1e184963"
server_key_iv: "43212eadd25c26202c1de0d0213f0f20"
server_key_version: "encryptionkey@0"
I20260926 13:09:04.834965 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:04.852249 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:04.854467 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:04.854596 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:04.855245 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:04.855294 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:04.855322 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:04.855356 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:09:04 dist-test-slave-2dw5 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:04.862046 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45199
I20260926 13:09:04.862402 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:09:04.862419 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45199 every 8 connection(s)
I20260926 13:09:04.864737 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:04.855684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37681 (local address 127.0.98.62:44863)
0926 13:09:04.855751 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:04.855759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:04.855946 (+ 187us) server_negotiation.cc:407] Connection header received
0926 13:09:04.856628 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:04.856629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:04.856641 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:04.856699 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:04.857542 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.858035 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.859117 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.859337 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.859404 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:04.862110 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:04.862112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:04.862112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:04.862120 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:04.863612 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:04.864003 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:04.864003 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:04.864004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:04.864090 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:04.864278 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:04.864278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:04.864278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:04.864378 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:04.864420 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:04.864617 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 13:09:04.864674 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:09:04.864917 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:04.855611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44863 (local address 127.0.98.2:37681)
0926 13:09:04.855809 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:04.855858 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:04.856499 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:04.856782 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:04.856787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:04.857143 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:04.857426 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:04.857432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.858130 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.858131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:04.859024 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:04.859027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.859423 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.859424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:04.859530 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:04.860001 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:04.860009 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:04.861958 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:04.863689 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:04.863694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:04.863696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:04.863910 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:04.864141 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:04.864141 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:04.864141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:04.864233 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:04.864412 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:04.864415 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:04.864525 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 13:09:04.864597 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 13:09:04.864714 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":120,"threads_started":1}
I20260926 13:09:04.865372 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44863
I20260926 13:09:04.865443 2465 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:04.865602 2465 heartbeater.cc:511] Master 127.0.98.62:44863 requested a full tablet report, sending...
I20260926 13:09:04.865972 2121 ts_manager.cc:194] Registered new tserver with Master: b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2:45199)
I20260926 13:09:04.866369 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37681
I20260926 13:09:04.869539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 2334
I20260926 13:09:04.869721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:04.869886 392 external_mini_cluster.cc:1428] Setting key 9e537ae554264f98ba28862434326349
I20260926 13:09:04.871829 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:04.866393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58855 (local address 127.0.98.62:44863)
0926 13:09:04.866482 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:04.866484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:04.866491 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:09:04.866555 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:04.866556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:04.866565 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:04.866618 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:04.866921 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.867467 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.868202 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.868375 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.868424 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:04.869264 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:04.869265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:04.869266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:04.869272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:04.870806 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:04.871121 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:04.871122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:04.871122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:04.871174 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:04.871331 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:04.871332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:04.871332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:04.871476 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:04.871533 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:04.871665 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 13:09:04.871737 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 13:09:04.871845 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:04.866319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44863 (local address 127.0.98.2:58855)
0926 13:09:04.866440 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:04.866450 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:04.866502 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:04.866621 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:04.866622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:04.866767 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:04.866819 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:04.866821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.867584 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.867585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:04.868121 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:04.868123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:04.868452 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:04.868452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:04.868515 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:04.868910 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:04.868914 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:04.869187 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:04.870892 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:04.870893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:04.870894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:04.871062 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:04.871226 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:04.871226 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:04.871227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:04.871283 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:04.871533 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:04.871533 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:04.871577 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 13:09:04.871638 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 13:09:04.871739 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:09:04.895002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44863
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:04.976402 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:04.976620 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:04.976648 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:04.977823 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:04.977862 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:04.977882 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 13:09:04.977893 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:09:04.977909 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:04.979423 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:04.979674 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:04.979862 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:04.984884 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:04.984954 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:04.985023 2474 server_base.cc:1034] running on GCE node
W20260926 13:09:04.984954 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:04.985277 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:04.985512 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:04.986670 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790428144986653 us; error 29 us; skew 500 ppm
Sep 26 13:09:04 dist-test-slave-2dw5 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:04.989478 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:09:04.990015 2474 webserver.cc:460] Webserver started at http://127.0.98.3:46537/ using document root <none> and password file <none>
I20260926 13:09:04.990226 2474 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:04.990274 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:04.990403 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:04.991588 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "1c1431a6e1614401bbf936a4f644d0cc"
format_stamp: "Formatted at 2026-09-26 13:09:04 on dist-test-slave-2dw5"
server_key: "dfaf04e655c5d9cf97745acdd5d7eef6"
server_key_iv: "d1acc26ddf5f02e4839435ed8c704846"
server_key_version: "encryptionkey@0"
I20260926 13:09:04.991920 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1c1431a6e1614401bbf936a4f644d0cc"
format_stamp: "Formatted at 2026-09-26 13:09:04 on dist-test-slave-2dw5"
server_key: "dfaf04e655c5d9cf97745acdd5d7eef6"
server_key_iv: "d1acc26ddf5f02e4839435ed8c704846"
server_key_version: "encryptionkey@0"
I20260926 13:09:04.993360 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:04.994323 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:04.994494 2474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:04.994668 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "1c1431a6e1614401bbf936a4f644d0cc"
format_stamp: "Formatted at 2026-09-26 13:09:04 on dist-test-slave-2dw5"
server_key: "dfaf04e655c5d9cf97745acdd5d7eef6"
server_key_iv: "d1acc26ddf5f02e4839435ed8c704846"
server_key_version: "encryptionkey@0"
I20260926 13:09:04.994737 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:05.016484 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:05.018797 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:05.018975 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:05.019516 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:05.019573 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:05.019606 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:05.019624 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:09:05 dist-test-slave-2dw5 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:05.026175 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33543
I20260926 13:09:05.026222 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33543 every 8 connection(s)
I20260926 13:09:05.026520 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:05.029304 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.020062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38691 (local address 127.0.98.62:44863)
0926 13:09:05.020112 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.020114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.020629 (+ 515us) server_negotiation.cc:407] Connection header received
0926 13:09:05.021126 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.021128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.021141 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.021217 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:05.022045 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.022862 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.023861 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.024067 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.024133 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.026509 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:05.026510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:05.026511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:05.026517 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:05.027890 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:05.028321 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:05.028323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:05.028323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:05.028378 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:05.028652 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:05.028653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:05.028654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:05.028774 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:05.028833 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.029120 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.029212 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:09:05.029462 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.020126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44863 (local address 127.0.98.3:38691)
0926 13:09:05.020328 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.020367 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.021001 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.021296 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.021300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.021663 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:05.021960 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.021966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.022944 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.022946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.023774 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.023776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.024167 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.024168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.024290 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.024901 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:05.024916 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:05.026402 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:05.027987 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:05.027993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:05.027996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:05.028248 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:05.028433 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:05.028433 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:05.028434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:05.028542 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:05.028835 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:05.028841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:05.028980 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.029080 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.029241 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260926 13:09:05.029969 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44863
I20260926 13:09:05.030026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 2474
I20260926 13:09:05.030076 2605 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:05.030104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:05.030272 2605 heartbeater.cc:511] Master 127.0.98.62:44863 requested a full tablet report, sending...
I20260926 13:09:05.030275 392 external_mini_cluster.cc:1428] Setting key f5852ecc7feff3e5bd5e70e7fffdc4dc
I20260926 13:09:05.030719 2121 ts_manager.cc:194] Registered new tserver with Master: 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3:33543)
I20260926 13:09:05.031126 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:09:05.031221 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38691
I20260926 13:09:05.036947 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.031064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55291 (local address 127.0.98.62:44863)
0926 13:09:05.031102 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.031104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.031209 (+ 105us) server_negotiation.cc:407] Connection header received
0926 13:09:05.031285 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.031286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.031295 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.031331 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:05.031635 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.032102 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.032953 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.033282 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.033324 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.034369 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:05.034371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:05.034371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:05.034378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:05.035883 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:05.036187 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:05.036188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:05.036188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:05.036240 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:05.036395 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:05.036396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:05.036396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:05.036530 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:05.036605 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.036786 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.036878 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 13:09:05.036952 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.031035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44863 (local address 127.0.98.3:55291)
0926 13:09:05.031154 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.031163 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.031232 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.031344 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.031345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.031511 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:05.031567 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.031568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.032187 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.032188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.032857 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.032859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.033404 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.033405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.033471 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.033897 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:05.033902 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:05.034287 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:05.035964 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:05.035965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:05.035965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:05.036126 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:05.036292 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:05.036292 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:05.036293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:05.036345 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:05.036595 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:05.036596 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:05.036640 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.036710 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.036839 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":75,"threads_started":1}
Sep 26 13:09:05 dist-test-slave-2dw5 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:09:05 dist-test-slave-2dw5 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:05.053720 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.046903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.0.98.62:44863)
0926 13:09:05.046962 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.046964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.046970 (+ 6us) server_negotiation.cc:407] Connection header received
0926 13:09:05.046992 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.046992 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.047001 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.047051 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:05.047696 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.048184 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.048957 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.049143 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.049197 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.051466 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:05.051468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:05.051468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:05.051475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:05.052709 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:05.053044 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:05.053044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:05.053045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:05.053091 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:05.053258 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:05.053259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:05.053259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:05.053361 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:05.053447 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.053551 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.053626 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:09:05.055289 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:05.056900 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:05.063205 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.059115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45199 (local address 127.0.0.1:47216)
0926 13:09:05.059472 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.059516 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.059718 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.060110 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.060111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.060113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:05.060143 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.060144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.061337 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.061338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.062240 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.062243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.062894 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.062895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.062974 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.062988 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.063050 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.063103 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260926 13:09:05.063210 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.059293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.0.98.2:45199)
0926 13:09:05.059445 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.059451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.059666 (+ 215us) server_negotiation.cc:407] Connection header received
0926 13:09:05.059966 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.059969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.059985 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.060051 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:05.060212 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.061270 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.062315 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.062801 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.062884 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.062949 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.063077 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.063110 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260926 13:09:05.063486 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.059120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40577 (local address 127.0.0.1:49744)
0926 13:09:05.059500 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.059516 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.059584 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.059781 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.059784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.059787 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:05.059864 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.059866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.061173 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.061174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.062252 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.062254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.063260 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.063261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.063320 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.063325 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.063373 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.063414 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":71,"threads_started":1}
I20260926 13:09:05.063676 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.059206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33543 (local address 127.0.0.1:60442)
0926 13:09:05.059807 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.059815 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.059850 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.060034 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.060035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.060037 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:05.060086 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.060088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.060948 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.060949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.062248 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.062251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.063399 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.063401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.063461 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.063464 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.063518 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.063620 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260926 13:09:05.063726 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.059364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.0.98.1:40577)
0926 13:09:05.059601 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.059604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.059614 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:05.059656 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.059658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.059675 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.059774 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:05.059925 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.061093 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.062348 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.063114 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.063175 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.063227 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.063597 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.063637 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260926 13:09:05.063983 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.059350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.0.98.3:33543)
0926 13:09:05.059901 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.059905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.059913 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:05.059947 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.059948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.059964 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.060031 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:05.060157 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.060868 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.062348 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.063110 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.063175 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.063226 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.063846 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.063885 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":65,"threads_started":1}
I20260926 13:09:05.064005 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 3af1302bbb82466a8494cb75fdb4ef18 (DEFAULT_TABLE table=test-table [id=f9d11a033cb74c04905f880f4b693a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:05.064389 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af1302bbb82466a8494cb75fdb4ef18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:05.064335 2242 tablet_service.cc:1467] Processing CreateTablet for tablet 3af1302bbb82466a8494cb75fdb4ef18 (DEFAULT_TABLE table=test-table [id=f9d11a033cb74c04905f880f4b693a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:05.064522 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 3af1302bbb82466a8494cb75fdb4ef18 (DEFAULT_TABLE table=test-table [id=f9d11a033cb74c04905f880f4b693a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:05.064639 2242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af1302bbb82466a8494cb75fdb4ef18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:05.064816 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af1302bbb82466a8494cb75fdb4ef18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:05.067332 2628 tablet_bootstrap.cc:492] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc: Bootstrap starting.
I20260926 13:09:05.068046 2629 tablet_bootstrap.cc:492] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9: Bootstrap starting.
I20260926 13:09:05.068202 2630 tablet_bootstrap.cc:492] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49: Bootstrap starting.
I20260926 13:09:05.068385 2628 tablet_bootstrap.cc:654] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:05.068650 2629 tablet_bootstrap.cc:654] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:05.068686 2628 log.cc:824] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:05.068743 2630 tablet_bootstrap.cc:654] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:05.068969 2629 log.cc:824] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:05.069041 2630 log.cc:824] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:05.069512 2628 tablet_bootstrap.cc:492] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc: No bootstrap required, opened a new log
I20260926 13:09:05.069585 2628 ts_tablet_manager.cc:1397] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:09:05.069646 2629 tablet_bootstrap.cc:492] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9: No bootstrap required, opened a new log
I20260926 13:09:05.069706 2629 ts_tablet_manager.cc:1397] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:05.069748 2630 tablet_bootstrap.cc:492] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49: No bootstrap required, opened a new log
I20260926 13:09:05.069808 2630 ts_tablet_manager.cc:1397] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:05.071430 2630 raft_consensus.cc:348] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.071579 2630 raft_consensus.cc:374] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:05.071611 2630 raft_consensus.cc:729] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04abd56465fe4d14b2c1056ea7649d49, State: Initialized, Role: FOLLOWER
I20260926 13:09:05.071610 2628 raft_consensus.cc:348] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.071610 2629 raft_consensus.cc:348] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.071727 2630 consensus_queue.cc:260] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.071782 2628 raft_consensus.cc:374] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:05.071779 2629 raft_consensus.cc:374] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:05.071820 2629 raft_consensus.cc:729] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d1a6afd64d42f8802cf1950d495fd9, State: Initialized, Role: FOLLOWER
I20260926 13:09:05.071820 2628 raft_consensus.cc:729] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1431a6e1614401bbf936a4f644d0cc, State: Initialized, Role: FOLLOWER
I20260926 13:09:05.071924 2629 consensus_queue.cc:260] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.071962 2630 ts_tablet_manager.cc:1428] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:09:05.071942 2628 consensus_queue.cc:260] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.072069 2325 heartbeater.cc:503] Master 127.0.98.62:44863 was elected leader, sending a full tablet report...
I20260926 13:09:05.072182 2629 ts_tablet_manager.cc:1428] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 13:09:05.072182 2628 ts_tablet_manager.cc:1428] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:09:05.072242 2465 heartbeater.cc:503] Master 127.0.98.62:44863 was elected leader, sending a full tablet report...
I20260926 13:09:05.072294 2605 heartbeater.cc:503] Master 127.0.98.62:44863 was elected leader, sending a full tablet report...
W20260926 13:09:05.113101 2466 tablet.cc:2369] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:09:05.127550 2636 raft_consensus.cc:482] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:05.127676 2636 raft_consensus.cc:504] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.127967 2636 leader_election.cc:288] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2:45199), 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577)
I20260926 13:09:05.131721 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.128246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51323 (local address 127.0.98.1:40577)
0926 13:09:05.128287 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.128289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.128382 (+ 93us) server_negotiation.cc:407] Connection header received
0926 13:09:05.128407 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.128408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.128419 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.128467 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:05.128592 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.129380 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.130784 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.131370 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.131435 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.131446 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.131604 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.131648 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:09:05.131726 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.128154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40577 (local address 127.0.98.3:51323)
0926 13:09:05.128313 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.128322 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.128372 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.128468 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.128469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.128469 (+ 0us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:05.128533 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.128535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.129465 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.129466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.130687 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.130689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.131448 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.131448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.131530 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.131535 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.131590 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.131630 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":19,"thread_start_us":102,"threads_started":1}
I20260926 13:09:05.131783 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.128199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57695 (local address 127.0.98.2:45199)
0926 13:09:05.128238 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.128240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.128251 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:05.128352 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.128358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.128370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.128432 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:05.128698 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.129384 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.130775 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.131370 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.131437 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.131446 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.131685 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.131724 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:09:05.131729 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.128152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45199 (local address 127.0.98.3:57695)
0926 13:09:05.128214 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.128225 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.128267 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.128449 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.128450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.128451 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:05.128583 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.128584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.129465 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.129466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.130686 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.130689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.131446 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.131448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.131583 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.131586 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.131638 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.131676 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":16}
I20260926 13:09:05.132292 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af1302bbb82466a8494cb75fdb4ef18" candidate_uuid: "1c1431a6e1614401bbf936a4f644d0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" is_pre_election: true
I20260926 13:09:05.132340 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af1302bbb82466a8494cb75fdb4ef18" candidate_uuid: "1c1431a6e1614401bbf936a4f644d0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04abd56465fe4d14b2c1056ea7649d49" is_pre_election: true
I20260926 13:09:05.132445 2419 raft_consensus.cc:2454] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c1431a6e1614401bbf936a4f644d0cc in term 0.
I20260926 13:09:05.132463 2279 raft_consensus.cc:2454] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c1431a6e1614401bbf936a4f644d0cc in term 0.
I20260926 13:09:05.132676 2493 leader_election.cc:302] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c1431a6e1614401bbf936a4f644d0cc, b4d1a6afd64d42f8802cf1950d495fd9; no voters:
I20260926 13:09:05.132828 2636 raft_consensus.cc:2781] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:05.132889 2636 raft_consensus.cc:482] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:05.132912 2636 raft_consensus.cc:3037] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:05.133591 2636 raft_consensus.cc:504] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.133711 2636 leader_election.cc:288] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [CANDIDATE]: Term 1 election: Requested vote from peers b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2:45199), 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577)
I20260926 13:09:05.133864 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af1302bbb82466a8494cb75fdb4ef18" candidate_uuid: "1c1431a6e1614401bbf936a4f644d0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04abd56465fe4d14b2c1056ea7649d49"
I20260926 13:09:05.133893 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af1302bbb82466a8494cb75fdb4ef18" candidate_uuid: "1c1431a6e1614401bbf936a4f644d0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4d1a6afd64d42f8802cf1950d495fd9"
I20260926 13:09:05.133955 2419 raft_consensus.cc:3037] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:05.133947 2279 raft_consensus.cc:3037] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:05.134619 2419 raft_consensus.cc:2454] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c1431a6e1614401bbf936a4f644d0cc in term 1.
I20260926 13:09:05.134701 2279 raft_consensus.cc:2454] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c1431a6e1614401bbf936a4f644d0cc in term 1.
I20260926 13:09:05.134773 2493 leader_election.cc:302] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c1431a6e1614401bbf936a4f644d0cc, b4d1a6afd64d42f8802cf1950d495fd9; no voters:
I20260926 13:09:05.134895 2636 raft_consensus.cc:2781] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:05.135075 2636 raft_consensus.cc:686] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 LEADER]: Becoming Leader. State: Replica: 1c1431a6e1614401bbf936a4f644d0cc, State: Running, Role: LEADER
I20260926 13:09:05.135171 2636 consensus_queue.cc:237] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.136168 2121 catalog_manager.cc:5495] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1c1431a6e1614401bbf936a4f644d0cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } health_report { overall_health: HEALTHY } } }
Sep 26 13:09:05 dist-test-slave-2dw5 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:05.186358 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.179762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34494 (local address 127.0.98.62:44863)
0926 13:09:05.179846 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.179848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.179935 (+ 87us) server_negotiation.cc:407] Connection header received
0926 13:09:05.180041 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.180041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.180049 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.180094 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:05.180577 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.181054 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.181833 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.182013 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.182066 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.184032 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:05.184033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:05.184034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:05.184041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:05.185265 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:05.185682 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:05.185683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:05.185683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:05.185729 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:05.185881 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:05.185882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:05.185882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:05.185989 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:05.186043 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.186175 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.186254 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:09:05.186430 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.179675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44863 (local address 127.0.0.1:34494)
0926 13:09:05.179861 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.179872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.179943 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.180105 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.180106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.180442 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:05.180499 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.180502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.181121 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.181123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.181752 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.181755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.182068 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.182069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.182115 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.182614 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:05.182618 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:05.183935 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:05.185335 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:05.185344 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:05.185346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:05.185615 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:05.185779 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:05.185779 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:05.185780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:05.185836 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:05.186040 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:05.186047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:05.186092 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.186150 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.186319 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":68,"thread_start_us":93,"threads_started":1}
I20260926 13:09:05.188516 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34494:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 13:09:05.188731 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:05.190078 2242 tablet_service.cc:1467] Processing CreateTablet for tablet cb08ec02ce594817b13eedbff1f592ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=333ffef245ce4985adda3bbb342e3a58]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:09:05.190130 2539 tablet_service.cc:1467] Processing CreateTablet for tablet cb08ec02ce594817b13eedbff1f592ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=333ffef245ce4985adda3bbb342e3a58]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:09:05.190115 2399 tablet_service.cc:1467] Processing CreateTablet for tablet cb08ec02ce594817b13eedbff1f592ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=333ffef245ce4985adda3bbb342e3a58]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:09:05.190260 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb08ec02ce594817b13eedbff1f592ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:05.190263 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb08ec02ce594817b13eedbff1f592ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:05.190274 2242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb08ec02ce594817b13eedbff1f592ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:05.191689 2629 tablet_bootstrap.cc:492] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9: Bootstrap starting.
I20260926 13:09:05.191730 2630 tablet_bootstrap.cc:492] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49: Bootstrap starting.
I20260926 13:09:05.192293 2630 tablet_bootstrap.cc:654] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:05.192297 2629 tablet_bootstrap.cc:654] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:05.192581 2628 tablet_bootstrap.cc:492] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: Bootstrap starting.
I20260926 13:09:05.192878 2630 tablet_bootstrap.cc:492] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49: No bootstrap required, opened a new log
I20260926 13:09:05.192927 2630 ts_tablet_manager.cc:1397] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:05.193071 2628 tablet_bootstrap.cc:654] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:05.193066 2630 raft_consensus.cc:348] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.193122 2630 raft_consensus.cc:374] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:05.193149 2630 raft_consensus.cc:729] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04abd56465fe4d14b2c1056ea7649d49, State: Initialized, Role: FOLLOWER
I20260926 13:09:05.193194 2630 consensus_queue.cc:260] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.193277 2630 ts_tablet_manager.cc:1428] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:05.193369 2634 tablet_replica.cc:439] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } }
I20260926 13:09:05.193444 2634 tablet_replica.cc:442] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:05.193522 2629 tablet_bootstrap.cc:492] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9: No bootstrap required, opened a new log
I20260926 13:09:05.193572 2629 ts_tablet_manager.cc:1397] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:05.193704 2628 tablet_bootstrap.cc:492] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: No bootstrap required, opened a new log
I20260926 13:09:05.193746 2628 ts_tablet_manager.cc:1397] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:05.193732 2629 raft_consensus.cc:348] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.193799 2629 raft_consensus.cc:374] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:05.193819 2629 raft_consensus.cc:729] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d1a6afd64d42f8802cf1950d495fd9, State: Initialized, Role: FOLLOWER
I20260926 13:09:05.193862 2629 consensus_queue.cc:260] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.193862 2628 raft_consensus.cc:348] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.193935 2628 raft_consensus.cc:374] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:05.193950 2629 ts_tablet_manager.cc:1428] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:05.193961 2628 raft_consensus.cc:729] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1431a6e1614401bbf936a4f644d0cc, State: Initialized, Role: FOLLOWER
I20260926 13:09:05.193998 2628 consensus_queue.cc:260] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.194082 2628 ts_tablet_manager.cc:1428] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:05.194095 2635 tablet_replica.cc:439] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } }
I20260926 13:09:05.194166 2635 tablet_replica.cc:442] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:05.194212 2636 tablet_replica.cc:439] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } }
I20260926 13:09:05.194298 2636 tablet_replica.cc:442] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 13:09:05.204941 2326 tablet.cc:2369] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:09:05.277146 2606 tablet.cc:2369] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:09:05.360154 2636 raft_consensus.cc:482] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:05.360265 2636 raft_consensus.cc:504] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.360435 2636 leader_election.cc:288] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577), b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2:45199)
I20260926 13:09:05.360659 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb08ec02ce594817b13eedbff1f592ad" candidate_uuid: "1c1431a6e1614401bbf936a4f644d0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04abd56465fe4d14b2c1056ea7649d49" is_pre_election: true
I20260926 13:09:05.360767 2279 raft_consensus.cc:2454] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c1431a6e1614401bbf936a4f644d0cc in term 0.
I20260926 13:09:05.360796 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb08ec02ce594817b13eedbff1f592ad" candidate_uuid: "1c1431a6e1614401bbf936a4f644d0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" is_pre_election: true
I20260926 13:09:05.360893 2419 raft_consensus.cc:2454] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c1431a6e1614401bbf936a4f644d0cc in term 0.
I20260926 13:09:05.361016 2492 leader_election.cc:302] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04abd56465fe4d14b2c1056ea7649d49, 1c1431a6e1614401bbf936a4f644d0cc; no voters:
I20260926 13:09:05.361128 2636 raft_consensus.cc:2781] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:05.361171 2636 raft_consensus.cc:482] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:05.361189 2636 raft_consensus.cc:3037] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:05.361851 2636 raft_consensus.cc:504] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.361991 2636 leader_election.cc:288] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [CANDIDATE]: Term 1 election: Requested vote from peers 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577), b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2:45199)
I20260926 13:09:05.362178 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb08ec02ce594817b13eedbff1f592ad" candidate_uuid: "1c1431a6e1614401bbf936a4f644d0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04abd56465fe4d14b2c1056ea7649d49"
I20260926 13:09:05.362176 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb08ec02ce594817b13eedbff1f592ad" candidate_uuid: "1c1431a6e1614401bbf936a4f644d0cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4d1a6afd64d42f8802cf1950d495fd9"
I20260926 13:09:05.362262 2419 raft_consensus.cc:3037] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:05.362263 2279 raft_consensus.cc:3037] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:05.363018 2279 raft_consensus.cc:2454] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c1431a6e1614401bbf936a4f644d0cc in term 1.
I20260926 13:09:05.363020 2419 raft_consensus.cc:2454] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c1431a6e1614401bbf936a4f644d0cc in term 1.
I20260926 13:09:05.363175 2492 leader_election.cc:302] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04abd56465fe4d14b2c1056ea7649d49, 1c1431a6e1614401bbf936a4f644d0cc; no voters:
I20260926 13:09:05.363279 2636 raft_consensus.cc:2781] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:05.363329 2636 raft_consensus.cc:686] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [term 1 LEADER]: Becoming Leader. State: Replica: 1c1431a6e1614401bbf936a4f644d0cc, State: Running, Role: LEADER
I20260926 13:09:05.363380 2636 consensus_queue.cc:237] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.363709 2638 tablet_replica.cc:439] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: TxnStatusTablet state changed. Reason: New leader 1c1431a6e1614401bbf936a4f644d0cc. Latest consensus state: current_term: 1 leader_uuid: "1c1431a6e1614401bbf936a4f644d0cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } }
I20260926 13:09:05.363788 2638 tablet_replica.cc:442] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:09:05.364024 2122 catalog_manager.cc:5495] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1c1431a6e1614401bbf936a4f644d0cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } health_report { overall_health: HEALTHY } } }
I20260926 13:09:05.364341 2650 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 13:09:05.364460 2650 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 13:09:05.377799 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.375003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33543 (local address 127.0.0.1:60450)
0926 13:09:05.375082 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.375095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.375159 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.375349 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.375350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.375352 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:05.375407 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.375409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.376317 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.376318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.376892 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.376894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.377333 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.377334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.377384 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.377399 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:05.377671 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:05.377675 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.377699 (+ 24us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.377737 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 13:09:05.377810 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.375124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.0.98.3:33543)
0926 13:09:05.375161 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.375163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.375172 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:05.375235 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.375236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.375256 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.375325 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:05.375483 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.376242 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.376963 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.377265 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.377330 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.377480 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:05.377616 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:05.377692 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.377709 (+ 17us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.377731 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:09:05.379528 2419 raft_consensus.cc:1261] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Refusing update from remote peer 1c1431a6e1614401bbf936a4f644d0cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:05.379554 2279 raft_consensus.cc:1261] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Refusing update from remote peer 1c1431a6e1614401bbf936a4f644d0cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:05.379918 2638 consensus_queue.cc:1035] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:05.380070 2636 consensus_queue.cc:1035] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:05.381208 2634 tablet_replica.cc:439] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49: TxnStatusTablet state changed. Reason: New leader 1c1431a6e1614401bbf936a4f644d0cc. Latest consensus state: current_term: 1 leader_uuid: "1c1431a6e1614401bbf936a4f644d0cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } }
I20260926 13:09:05.381307 2634 tablet_replica.cc:442] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:05.381659 2635 tablet_replica.cc:439] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9: TxnStatusTablet state changed. Reason: New leader 1c1431a6e1614401bbf936a4f644d0cc. Latest consensus state: current_term: 1 leader_uuid: "1c1431a6e1614401bbf936a4f644d0cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } }
I20260926 13:09:05.381747 2635 tablet_replica.cc:442] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:05.382419 2638 tablet_replica.cc:439] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c1431a6e1614401bbf936a4f644d0cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } }
I20260926 13:09:05.382506 2638 tablet_replica.cc:442] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:09:05.383920 2636 tablet_replica.cc:439] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c1431a6e1614401bbf936a4f644d0cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } }
I20260926 13:09:05.384002 2636 tablet_replica.cc:442] T cb08ec02ce594817b13eedbff1f592ad P 1c1431a6e1614401bbf936a4f644d0cc: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:09:05.384203 2635 tablet_replica.cc:439] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c1431a6e1614401bbf936a4f644d0cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } }
I20260926 13:09:05.384281 2635 tablet_replica.cc:442] T cb08ec02ce594817b13eedbff1f592ad P b4d1a6afd64d42f8802cf1950d495fd9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:05.384528 2634 tablet_replica.cc:439] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c1431a6e1614401bbf936a4f644d0cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } } }
I20260926 13:09:05.384624 2634 tablet_replica.cc:442] T cb08ec02ce594817b13eedbff1f592ad P 04abd56465fe4d14b2c1056ea7649d49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:05.385169 2655 mvcc.cc:205] Tried to move back new op lower bound from 7333593683471908864 to 7333593683408564224. Current Snapshot: MvccSnapshot[applied={T|T < 7333593683471908864}]
I20260926 13:09:05.388183 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.385540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.0.98.3:33543)
0926 13:09:05.385579 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.385581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.385612 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:09:05.385680 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.385680 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.385688 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.385727 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:05.385833 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.386287 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.387411 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.387712 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.387784 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.387964 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:05.387996 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:05.388046 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.388111 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.388130 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:09:05.393718 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.390742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42741 (local address 127.0.98.3:33543)
0926 13:09:05.390774 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.390775 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.390783 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:05.390871 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.390872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.390881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.390924 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:05.391040 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.391531 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.392775 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.392947 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.393005 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.393310 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:05.393359 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:05.393424 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.393585 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.393606 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 13:09:05.393790 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.390686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33543 (local address 127.0.98.3:42741)
0926 13:09:05.390742 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.390753 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.390795 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.390928 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.390928 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.390929 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:05.390986 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.390988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.391602 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.391603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.392695 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.392697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.393010 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.393011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.393077 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.393096 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:05.393431 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:05.393432 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.393578 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.393622 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 13:09:05.396358 2279 raft_consensus.cc:1261] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Refusing update from remote peer 1c1431a6e1614401bbf936a4f644d0cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:05.396358 2419 raft_consensus.cc:1261] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Refusing update from remote peer 1c1431a6e1614401bbf936a4f644d0cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:05.396600 2660 consensus_queue.cc:1035] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:05.396745 2638 consensus_queue.cc:1035] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:05.397905 2653 mvcc.cc:205] Tried to move back new op lower bound from 7333593683541884928 to 7333593682474311680. Current Snapshot: MvccSnapshot[applied={T|T < 7333593683541884928}]
I20260926 13:09:05.398653 2654 mvcc.cc:205] Tried to move back new op lower bound from 7333593683541884928 to 7333593682474311680. Current Snapshot: MvccSnapshot[applied={T|T < 7333593683541884928}]
I20260926 13:09:05.411527 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:05.411651 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:05.412487 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7: Sending DeleteTablet for 3 replicas of tablet 3af1302bbb82466a8494cb75fdb4ef18
I20260926 13:09:05.412845 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af1302bbb82466a8494cb75fdb4ef18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47216
I20260926 13:09:05.412891 2242 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af1302bbb82466a8494cb75fdb4ef18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49744
I20260926 13:09:05.413156 2668 tablet_replica.cc:330] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9: stopping tablet replica
I20260926 13:09:05.413190 2669 tablet_replica.cc:330] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49: stopping tablet replica
I20260926 13:09:05.413126 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:05.413272 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:05.413326 2669 raft_consensus.cc:2229] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:05.413385 2669 raft_consensus.cc:2258] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:05.413533 2668 raft_consensus.cc:2229] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:05.413781 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 3af1302bbb82466a8494cb75fdb4ef18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60442
I20260926 13:09:05.413859 2669 ts_tablet_manager.cc:1905] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:05.414247 2668 raft_consensus.cc:2258] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:05.414686 2399 tablet_service.cc:1467] Processing CreateTablet for tablet bb828620680846f7b29cb0c1f2f65d99 (DEFAULT_TABLE table=test-table [id=1dd08f3fc418468ba5d5b0299e0658e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:05.414722 2668 ts_tablet_manager.cc:1905] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:05.414911 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb828620680846f7b29cb0c1f2f65d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:05.415884 2669 ts_tablet_manager.cc:1918] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:05.415975 2669 log.cc:1196] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/wal/wals/3af1302bbb82466a8494cb75fdb4ef18
I20260926 13:09:05.416152 2629 tablet_bootstrap.cc:492] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9: Bootstrap starting.
I20260926 13:09:05.416316 2669 ts_tablet_manager.cc:1939] T 3af1302bbb82466a8494cb75fdb4ef18 P 04abd56465fe4d14b2c1056ea7649d49: Deleting consensus metadata
I20260926 13:09:05.416353 2668 ts_tablet_manager.cc:1918] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:05.416406 2668 log.cc:1196] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/3af1302bbb82466a8494cb75fdb4ef18
I20260926 13:09:05.416539 2629 tablet_bootstrap.cc:654] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:05.416613 2668 ts_tablet_manager.cc:1939] T 3af1302bbb82466a8494cb75fdb4ef18 P b4d1a6afd64d42f8802cf1950d495fd9: Deleting consensus metadata
I20260926 13:09:05.416888 2108 catalog_manager.cc:4841] TS 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577): tablet 3af1302bbb82466a8494cb75fdb4ef18 (table test-table [id=f9d11a033cb74c04905f880f4b693a9e]) successfully deleted
I20260926 13:09:05.417006 2242 tablet_service.cc:1467] Processing CreateTablet for tablet bb828620680846f7b29cb0c1f2f65d99 (DEFAULT_TABLE table=test-table [id=1dd08f3fc418468ba5d5b0299e0658e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:05.417124 2242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb828620680846f7b29cb0c1f2f65d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:05.417584 2670 tablet_replica.cc:330] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc: stopping tablet replica
I20260926 13:09:05.417624 2109 catalog_manager.cc:4841] TS b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2:45199): tablet 3af1302bbb82466a8494cb75fdb4ef18 (table test-table [id=f9d11a033cb74c04905f880f4b693a9e]) successfully deleted
I20260926 13:09:05.417779 2670 raft_consensus.cc:2229] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:09:05.417888 2670 raft_consensus.cc:2258] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:05.418251 2630 tablet_bootstrap.cc:492] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49: Bootstrap starting.
I20260926 13:09:05.418282 2629 tablet_bootstrap.cc:492] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9: No bootstrap required, opened a new log
I20260926 13:09:05.418322 2629 ts_tablet_manager.cc:1397] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:05.418460 2670 ts_tablet_manager.cc:1905] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:05.418462 2629 raft_consensus.cc:348] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.418529 2629 raft_consensus.cc:374] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:05.418547 2629 raft_consensus.cc:729] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d1a6afd64d42f8802cf1950d495fd9, State: Initialized, Role: FOLLOWER
I20260926 13:09:05.418586 2629 consensus_queue.cc:260] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.418634 2630 tablet_bootstrap.cc:654] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:05.418674 2629 ts_tablet_manager.cc:1428] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:05.419384 2538 tablet_service.cc:1467] Processing CreateTablet for tablet bb828620680846f7b29cb0c1f2f65d99 (DEFAULT_TABLE table=test-table [id=1dd08f3fc418468ba5d5b0299e0658e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:05.419492 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb828620680846f7b29cb0c1f2f65d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:05.420048 2670 ts_tablet_manager.cc:1918] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:05.420157 2670 log.cc:1196] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/3af1302bbb82466a8494cb75fdb4ef18
I20260926 13:09:05.420213 2630 tablet_bootstrap.cc:492] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49: No bootstrap required, opened a new log
I20260926 13:09:05.420259 2630 ts_tablet_manager.cc:1397] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:05.420396 2630 raft_consensus.cc:348] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.420470 2630 raft_consensus.cc:374] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:05.420490 2630 raft_consensus.cc:729] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04abd56465fe4d14b2c1056ea7649d49, State: Initialized, Role: FOLLOWER
I20260926 13:09:05.420531 2630 consensus_queue.cc:260] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.420616 2630 ts_tablet_manager.cc:1428] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:05.420707 2670 ts_tablet_manager.cc:1939] T 3af1302bbb82466a8494cb75fdb4ef18 P 1c1431a6e1614401bbf936a4f644d0cc: Deleting consensus metadata
I20260926 13:09:05.420781 2628 tablet_bootstrap.cc:492] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc: Bootstrap starting.
I20260926 13:09:05.421136 2628 tablet_bootstrap.cc:654] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:05.421962 2628 tablet_bootstrap.cc:492] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc: No bootstrap required, opened a new log
I20260926 13:09:05.422012 2628 ts_tablet_manager.cc:1397] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:05.422135 2628 raft_consensus.cc:348] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.422202 2628 raft_consensus.cc:374] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:05.422220 2628 raft_consensus.cc:729] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1431a6e1614401bbf936a4f644d0cc, State: Initialized, Role: FOLLOWER
I20260926 13:09:05.422272 2628 consensus_queue.cc:260] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.422358 2628 ts_tablet_manager.cc:1428] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:05.422561 2106 catalog_manager.cc:4841] TS 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3:33543): tablet 3af1302bbb82466a8494cb75fdb4ef18 (table test-table [id=f9d11a033cb74c04905f880f4b693a9e]) successfully deleted
I20260926 13:09:05.483735 2635 raft_consensus.cc:482] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:05.483829 2635 raft_consensus.cc:504] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.484143 2635 leader_election.cc:288] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577), 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3:33543)
I20260926 13:09:05.488091 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.484488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58521 (local address 127.0.98.1:40577)
0926 13:09:05.484531 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.484533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.484722 (+ 189us) server_negotiation.cc:407] Connection header received
0926 13:09:05.484810 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.484811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.484819 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.484846 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:05.485072 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.485805 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.487421 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.487778 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.487839 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.487851 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.487991 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.488025 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:09:05.488112 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.484329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40577 (local address 127.0.98.2:58521)
0926 13:09:05.484668 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.484677 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.484743 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.484897 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.484897 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.484902 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:05.484998 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.485000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.485904 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.485905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.487285 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.487289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.487851 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.487852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.487923 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.487927 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.487972 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.488018 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":120,"thread_start_us":61,"threads_started":1}
I20260926 13:09:05.488350 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.484493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50309 (local address 127.0.98.3:33543)
0926 13:09:05.484569 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.484571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.484604 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:09:05.484791 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.484792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.484803 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.484832 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:05.485072 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.485812 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.487396 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.487937 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.488026 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.488039 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.488244 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.488283 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:09:05.488534 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb828620680846f7b29cb0c1f2f65d99" candidate_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04abd56465fe4d14b2c1056ea7649d49" is_pre_election: true
I20260926 13:09:05.488590 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.484329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33543 (local address 127.0.98.2:50309)
0926 13:09:05.484543 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.484554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.484706 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.484895 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.484896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.484901 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:05.484998 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.485000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.485904 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.485905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.487285 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.487289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.488079 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.488080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.488163 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.488167 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.488446 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.488493 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":34,"spinlock_wait_cycles":2304,"thread_start_us":72,"threads_started":1}
I20260926 13:09:05.488638 2279 raft_consensus.cc:2454] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4d1a6afd64d42f8802cf1950d495fd9 in term 0.
I20260926 13:09:05.488845 2352 leader_election.cc:302] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04abd56465fe4d14b2c1056ea7649d49, b4d1a6afd64d42f8802cf1950d495fd9; no voters:
I20260926 13:09:05.488977 2635 raft_consensus.cc:2781] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:05.488919 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb828620680846f7b29cb0c1f2f65d99" candidate_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c1431a6e1614401bbf936a4f644d0cc" is_pre_election: true
I20260926 13:09:05.489018 2635 raft_consensus.cc:482] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:05.489037 2635 raft_consensus.cc:3037] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:05.489040 2559 raft_consensus.cc:2454] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4d1a6afd64d42f8802cf1950d495fd9 in term 0.
I20260926 13:09:05.489742 2635 raft_consensus.cc:504] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.489866 2635 leader_election.cc:288] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [CANDIDATE]: Term 1 election: Requested vote from peers 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577), 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3:33543)
I20260926 13:09:05.490007 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb828620680846f7b29cb0c1f2f65d99" candidate_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c1431a6e1614401bbf936a4f644d0cc"
I20260926 13:09:05.490023 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb828620680846f7b29cb0c1f2f65d99" candidate_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04abd56465fe4d14b2c1056ea7649d49"
I20260926 13:09:05.490088 2279 raft_consensus.cc:3037] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:05.490090 2559 raft_consensus.cc:3037] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:05.490653 2279 raft_consensus.cc:2454] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4d1a6afd64d42f8802cf1950d495fd9 in term 1.
I20260926 13:09:05.490809 2352 leader_election.cc:302] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04abd56465fe4d14b2c1056ea7649d49, b4d1a6afd64d42f8802cf1950d495fd9; no voters:
I20260926 13:09:05.490929 2635 raft_consensus.cc:2781] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:05.490954 2559 raft_consensus.cc:2454] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4d1a6afd64d42f8802cf1950d495fd9 in term 1.
I20260926 13:09:05.491077 2635 raft_consensus.cc:686] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 LEADER]: Becoming Leader. State: Replica: b4d1a6afd64d42f8802cf1950d495fd9, State: Running, Role: LEADER
I20260926 13:09:05.491142 2635 consensus_queue.cc:237] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:05.491853 2122 catalog_manager.cc:5495] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } health_report { overall_health: UNKNOWN } } }
I20260926 13:09:05.500121 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.497731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.0.98.2:45199)
0926 13:09:05.497769 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.497771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.497780 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:05.497855 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.497855 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.497872 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.497978 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:05.498134 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.498650 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.499401 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.499624 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.499679 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.499813 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:05.499917 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:05.499976 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.500022 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.500043 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:09:05.505309 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.502485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33543 (local address 127.0.98.2:59151)
0926 13:09:05.502695 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.502707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.502760 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.502890 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.502891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.502893 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:05.502963 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.502964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.503768 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.503769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.504346 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.504347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.504875 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.504876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.504929 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.504943 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:05.505136 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:05.505139 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.505189 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.505231 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260926 13:09:05.505296 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.502541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59151 (local address 127.0.98.3:33543)
0926 13:09:05.502630 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.502632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.502763 (+ 131us) server_negotiation.cc:407] Connection header received
0926 13:09:05.502801 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.502802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.502812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.502886 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:05.503011 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.503699 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.504426 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.504722 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.504836 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.505019 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:05.505051 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:05.505095 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.505197 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.505215 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 13:09:05.507277 2279 raft_consensus.cc:1261] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Refusing update from remote peer b4d1a6afd64d42f8802cf1950d495fd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:05.507316 2559 raft_consensus.cc:1261] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Refusing update from remote peer b4d1a6afd64d42f8802cf1950d495fd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:05.507593 2676 consensus_queue.cc:1035] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:05.507742 2635 consensus_queue.cc:1035] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:05.515681 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.512984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45199 (local address 127.0.98.3:43091)
0926 13:09:05.513020 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:05.513031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:05.513070 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:05.513228 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:05.513229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:05.513230 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:05.513277 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.513278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.514194 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.514195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.514911 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:05.514913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.515324 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.515325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:05.515381 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.515387 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:05.515530 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:05.515532 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:09:05.515570 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 13:09:05.515612 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 13:09:05.515676 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:05.513058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43091 (local address 127.0.98.2:45199)
0926 13:09:05.513119 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:05.513121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:05.513128 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:09:05.513149 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:05.513149 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:05.513162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:05.513213 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:05.513346 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.514102 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.514987 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:05.515260 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:05.515318 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:05.515446 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:05.515482 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:05.515521 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:05.515571 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 13:09:05.515591 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 13:09:05.516038 2654 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 13:09:07.041752 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:07.041869 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:07.042744 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7: Sending DeleteTablet for 3 replicas of tablet bb828620680846f7b29cb0c1f2f65d99
I20260926 13:09:07.043031 2242 tablet_service.cc:1514] Processing DeleteTablet for tablet bb828620680846f7b29cb0c1f2f65d99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49744
I20260926 13:09:07.043031 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet bb828620680846f7b29cb0c1f2f65d99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47216
I20260926 13:09:07.043079 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet bb828620680846f7b29cb0c1f2f65d99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60442
I20260926 13:09:07.043268 2710 tablet_replica.cc:330] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc: stopping tablet replica
I20260926 13:09:07.043334 2710 raft_consensus.cc:2229] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:07.043371 2710 raft_consensus.cc:2258] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:07.045078 2710 ts_tablet_manager.cc:1905] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:07.047093 2709 tablet_replica.cc:330] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9: stopping tablet replica
I20260926 13:09:07.047158 2709 raft_consensus.cc:2229] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:09:07.047256 2709 raft_consensus.cc:2258] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:07.047333 2710 ts_tablet_manager.cc:1918] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:09:07.047408 2710 log.cc:1196] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/bb828620680846f7b29cb0c1f2f65d99
I20260926 13:09:07.047508 2709 ts_tablet_manager.cc:1905] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:07.047632 2710 ts_tablet_manager.cc:1939] T bb828620680846f7b29cb0c1f2f65d99 P 1c1431a6e1614401bbf936a4f644d0cc: Deleting consensus metadata
I20260926 13:09:07.047981 2106 catalog_manager.cc:4841] TS 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3:33543): tablet bb828620680846f7b29cb0c1f2f65d99 (table test-table [id=1dd08f3fc418468ba5d5b0299e0658e5]) successfully deleted
I20260926 13:09:07.049043 2709 ts_tablet_manager.cc:1918] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:09:07.049108 2709 log.cc:1196] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/bb828620680846f7b29cb0c1f2f65d99
I20260926 13:09:07.049302 2709 ts_tablet_manager.cc:1939] T bb828620680846f7b29cb0c1f2f65d99 P b4d1a6afd64d42f8802cf1950d495fd9: Deleting consensus metadata
I20260926 13:09:07.049445 2708 tablet_replica.cc:330] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49: stopping tablet replica
I20260926 13:09:07.049499 2708 raft_consensus.cc:2229] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:07.049543 2708 raft_consensus.cc:2258] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:07.049762 2708 ts_tablet_manager.cc:1905] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:07.049823 2109 catalog_manager.cc:4841] TS b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2:45199): tablet bb828620680846f7b29cb0c1f2f65d99 (table test-table [id=1dd08f3fc418468ba5d5b0299e0658e5]) successfully deleted
I20260926 13:09:07.051023 2708 ts_tablet_manager.cc:1918] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:09:07.051074 2708 log.cc:1196] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/wal/wals/bb828620680846f7b29cb0c1f2f65d99
I20260926 13:09:07.051252 2708 ts_tablet_manager.cc:1939] T bb828620680846f7b29cb0c1f2f65d99 P 04abd56465fe4d14b2c1056ea7649d49: Deleting consensus metadata
I20260926 13:09:07.051553 2108 catalog_manager.cc:4841] TS 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577): tablet bb828620680846f7b29cb0c1f2f65d99 (table test-table [id=1dd08f3fc418468ba5d5b0299e0658e5]) successfully deleted
Sep 26 13:09:07 dist-test-slave-2dw5 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:09:07 dist-test-slave-2dw5 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:07.065922 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:07.059252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.0.98.62:44863)
0926 13:09:07.059403 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:07.059405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:07.059414 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:07.059464 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:07.059464 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:07.059474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:07.059531 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:07.060046 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.060607 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.061299 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.061540 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.061598 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:07.063704 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:07.063705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:07.063705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:07.063711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:07.065080 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:07.065346 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:07.065347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:07.065347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:07.065429 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:07.065549 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:07.065549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:07.065549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:07.065642 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:07.065706 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:07.065777 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 13:09:07.065836 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260926 13:09:07.067414 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:07.067566 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:07.068722 2242 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf1256d51d54fb38f70805e6765b4e6 (DEFAULT_TABLE table=test-table [id=27c3c0df645d4f1ca40e29c119e272d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:07.068733 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf1256d51d54fb38f70805e6765b4e6 (DEFAULT_TABLE table=test-table [id=27c3c0df645d4f1ca40e29c119e272d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:07.068858 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf1256d51d54fb38f70805e6765b4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:07.068859 2242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf1256d51d54fb38f70805e6765b4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:07.068853 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf1256d51d54fb38f70805e6765b4e6 (DEFAULT_TABLE table=test-table [id=27c3c0df645d4f1ca40e29c119e272d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:07.068961 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf1256d51d54fb38f70805e6765b4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:07.070253 2719 tablet_bootstrap.cc:492] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49: Bootstrap starting.
I20260926 13:09:07.070604 2721 tablet_bootstrap.cc:492] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc: Bootstrap starting.
I20260926 13:09:07.070619 2719 tablet_bootstrap.cc:654] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:07.070945 2720 tablet_bootstrap.cc:492] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9: Bootstrap starting.
I20260926 13:09:07.071007 2721 tablet_bootstrap.cc:654] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:07.071246 2719 tablet_bootstrap.cc:492] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49: No bootstrap required, opened a new log
I20260926 13:09:07.071267 2720 tablet_bootstrap.cc:654] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:07.071297 2719 ts_tablet_manager.cc:1397] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:07.071429 2719 raft_consensus.cc:348] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.071502 2719 raft_consensus.cc:374] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:07.071521 2719 raft_consensus.cc:729] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04abd56465fe4d14b2c1056ea7649d49, State: Initialized, Role: FOLLOWER
I20260926 13:09:07.071555 2721 tablet_bootstrap.cc:492] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc: No bootstrap required, opened a new log
I20260926 13:09:07.071592 2721 ts_tablet_manager.cc:1397] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:07.071570 2719 consensus_queue.cc:260] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.071693 2721 raft_consensus.cc:348] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.071740 2719 ts_tablet_manager.cc:1428] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 13:09:07.071827 2721 raft_consensus.cc:374] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:07.071859 2721 raft_consensus.cc:729] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1431a6e1614401bbf936a4f644d0cc, State: Initialized, Role: FOLLOWER
I20260926 13:09:07.071925 2720 tablet_bootstrap.cc:492] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9: No bootstrap required, opened a new log
I20260926 13:09:07.071928 2721 consensus_queue.cc:260] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.072031 2720 ts_tablet_manager.cc:1397] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:07.072168 2720 raft_consensus.cc:348] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.072221 2721 ts_tablet_manager.cc:1428] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:07.072301 2720 raft_consensus.cc:374] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:07.072358 2720 raft_consensus.cc:729] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d1a6afd64d42f8802cf1950d495fd9, State: Initialized, Role: FOLLOWER
I20260926 13:09:07.072412 2720 consensus_queue.cc:260] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.072512 2720 ts_tablet_manager.cc:1428] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:07.108691 2699 raft_consensus.cc:482] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:07.108788 2699 raft_consensus.cc:504] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.108963 2699 leader_election.cc:288] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577), 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3:33543)
I20260926 13:09:07.109176 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf1256d51d54fb38f70805e6765b4e6" candidate_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c1431a6e1614401bbf936a4f644d0cc" is_pre_election: true
I20260926 13:09:07.109186 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf1256d51d54fb38f70805e6765b4e6" candidate_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04abd56465fe4d14b2c1056ea7649d49" is_pre_election: true
I20260926 13:09:07.109261 2559 raft_consensus.cc:2454] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4d1a6afd64d42f8802cf1950d495fd9 in term 0.
I20260926 13:09:07.109283 2279 raft_consensus.cc:2454] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4d1a6afd64d42f8802cf1950d495fd9 in term 0.
I20260926 13:09:07.109479 2352 leader_election.cc:302] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04abd56465fe4d14b2c1056ea7649d49, b4d1a6afd64d42f8802cf1950d495fd9; no voters:
I20260926 13:09:07.109594 2699 raft_consensus.cc:2781] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:07.109629 2699 raft_consensus.cc:482] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:07.109643 2699 raft_consensus.cc:3037] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:07.110355 2699 raft_consensus.cc:504] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.110481 2699 leader_election.cc:288] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [CANDIDATE]: Term 1 election: Requested vote from peers 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577), 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3:33543)
I20260926 13:09:07.110630 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf1256d51d54fb38f70805e6765b4e6" candidate_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04abd56465fe4d14b2c1056ea7649d49"
I20260926 13:09:07.110643 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf1256d51d54fb38f70805e6765b4e6" candidate_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c1431a6e1614401bbf936a4f644d0cc"
I20260926 13:09:07.110702 2279 raft_consensus.cc:3037] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:07.110719 2559 raft_consensus.cc:3037] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:07.111307 2279 raft_consensus.cc:2454] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4d1a6afd64d42f8802cf1950d495fd9 in term 1.
I20260926 13:09:07.111413 2559 raft_consensus.cc:2454] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4d1a6afd64d42f8802cf1950d495fd9 in term 1.
I20260926 13:09:07.111471 2352 leader_election.cc:302] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04abd56465fe4d14b2c1056ea7649d49, b4d1a6afd64d42f8802cf1950d495fd9; no voters:
I20260926 13:09:07.111549 2699 raft_consensus.cc:2781] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:07.111599 2699 raft_consensus.cc:686] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 LEADER]: Becoming Leader. State: Replica: b4d1a6afd64d42f8802cf1950d495fd9, State: Running, Role: LEADER
I20260926 13:09:07.111640 2699 consensus_queue.cc:237] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.112224 2121 catalog_manager.cc:5495] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } health_report { overall_health: UNKNOWN } } }
I20260926 13:09:07.120184 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:07.118019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.0.98.2:45199)
0926 13:09:07.118161 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:07.118163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:07.118169 (+ 6us) server_negotiation.cc:407] Connection header received
0926 13:09:07.118189 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:07.118190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:07.118199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:07.118252 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:07.118358 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.118865 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.119564 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.119782 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.119833 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:07.119960 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:07.119993 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:07.120035 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:07.120079 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 13:09:07.120100 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260926 13:09:07.122328 2559 raft_consensus.cc:1261] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Refusing update from remote peer b4d1a6afd64d42f8802cf1950d495fd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:07.122354 2279 raft_consensus.cc:1261] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Refusing update from remote peer b4d1a6afd64d42f8802cf1950d495fd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:07.122591 2699 consensus_queue.cc:1035] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:07.122704 2702 consensus_queue.cc:1035] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:07.132349 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:07.132480 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:07.133055 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7: Sending DeleteTablet for 3 replicas of tablet 6cf1256d51d54fb38f70805e6765b4e6
I20260926 13:09:07.133301 2242 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf1256d51d54fb38f70805e6765b4e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49744
I20260926 13:09:07.133306 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf1256d51d54fb38f70805e6765b4e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47216
I20260926 13:09:07.133392 2708 tablet_replica.cc:330] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49: stopping tablet replica
I20260926 13:09:07.133399 2709 tablet_replica.cc:330] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9: stopping tablet replica
I20260926 13:09:07.133364 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf1256d51d54fb38f70805e6765b4e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60442
I20260926 13:09:07.133440 2709 raft_consensus.cc:2229] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:09:07.133440 2708 raft_consensus.cc:2229] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:07.133462 2710 tablet_replica.cc:330] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc: stopping tablet replica
I20260926 13:09:07.133476 2708 raft_consensus.cc:2258] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:07.133490 2709 raft_consensus.cc:2258] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:07.133515 2710 raft_consensus.cc:2229] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:07.133569 2710 raft_consensus.cc:2258] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:07.133781 2709 ts_tablet_manager.cc:1905] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:07.133781 2708 ts_tablet_manager.cc:1905] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:07.133950 2710 ts_tablet_manager.cc:1905] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:07.133948 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:07.134087 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:07.135181 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 2704cc671b07462db7a38f0f984e1ace (DEFAULT_TABLE table=test-table [id=6263170e0e16421b95c505c558e03c52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:07.135479 2708 ts_tablet_manager.cc:1918] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:07.135492 2710 ts_tablet_manager.cc:1918] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:07.135505 2242 tablet_service.cc:1467] Processing CreateTablet for tablet 2704cc671b07462db7a38f0f984e1ace (DEFAULT_TABLE table=test-table [id=6263170e0e16421b95c505c558e03c52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:07.135531 2708 log.cc:1196] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-0/wal/wals/6cf1256d51d54fb38f70805e6765b4e6
I20260926 13:09:07.135540 2710 log.cc:1196] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/6cf1256d51d54fb38f70805e6765b4e6
I20260926 13:09:07.135555 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 2704cc671b07462db7a38f0f984e1ace (DEFAULT_TABLE table=test-table [id=6263170e0e16421b95c505c558e03c52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:07.135586 2242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2704cc671b07462db7a38f0f984e1ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:07.135636 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2704cc671b07462db7a38f0f984e1ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:07.135751 2708 ts_tablet_manager.cc:1939] T 6cf1256d51d54fb38f70805e6765b4e6 P 04abd56465fe4d14b2c1056ea7649d49: Deleting consensus metadata
I20260926 13:09:07.135762 2710 ts_tablet_manager.cc:1939] T 6cf1256d51d54fb38f70805e6765b4e6 P 1c1431a6e1614401bbf936a4f644d0cc: Deleting consensus metadata
I20260926 13:09:07.136171 2709 ts_tablet_manager.cc:1918] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:07.136197 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2704cc671b07462db7a38f0f984e1ace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:07.136224 2709 log.cc:1196] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/6cf1256d51d54fb38f70805e6765b4e6
I20260926 13:09:07.136417 2709 ts_tablet_manager.cc:1939] T 6cf1256d51d54fb38f70805e6765b4e6 P b4d1a6afd64d42f8802cf1950d495fd9: Deleting consensus metadata
I20260926 13:09:07.136654 2719 tablet_bootstrap.cc:492] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49: Bootstrap starting.
I20260926 13:09:07.136721 2721 tablet_bootstrap.cc:492] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc: Bootstrap starting.
I20260926 13:09:07.137009 2719 tablet_bootstrap.cc:654] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:07.137012 2721 tablet_bootstrap.cc:654] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:07.137348 2109 catalog_manager.cc:4841] TS b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2:45199): tablet 6cf1256d51d54fb38f70805e6765b4e6 (table test-table [id=27c3c0df645d4f1ca40e29c119e272d3]) successfully deleted
I20260926 13:09:07.137369 2720 tablet_bootstrap.cc:492] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9: Bootstrap starting.
I20260926 13:09:07.137537 2108 catalog_manager.cc:4841] TS 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577): tablet 6cf1256d51d54fb38f70805e6765b4e6 (table test-table [id=27c3c0df645d4f1ca40e29c119e272d3]) successfully deleted
I20260926 13:09:07.137663 2720 tablet_bootstrap.cc:654] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:07.137796 2721 tablet_bootstrap.cc:492] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc: No bootstrap required, opened a new log
I20260926 13:09:07.137847 2721 ts_tablet_manager.cc:1397] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:07.137918 2719 tablet_bootstrap.cc:492] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49: No bootstrap required, opened a new log
I20260926 13:09:07.137965 2719 ts_tablet_manager.cc:1397] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:07.137967 2721 raft_consensus.cc:348] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.138046 2721 raft_consensus.cc:374] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:07.138072 2721 raft_consensus.cc:729] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1431a6e1614401bbf936a4f644d0cc, State: Initialized, Role: FOLLOWER
I20260926 13:09:07.138113 2721 consensus_queue.cc:260] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.138105 2719 raft_consensus.cc:348] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.138168 2719 raft_consensus.cc:374] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:07.138191 2719 raft_consensus.cc:729] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04abd56465fe4d14b2c1056ea7649d49, State: Initialized, Role: FOLLOWER
I20260926 13:09:07.138216 2721 ts_tablet_manager.cc:1428] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:07.138224 2719 consensus_queue.cc:260] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.138302 2719 ts_tablet_manager.cc:1428] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 13:09:07.138325 2106 catalog_manager.cc:4841] TS 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3:33543): tablet 6cf1256d51d54fb38f70805e6765b4e6 (table test-table [id=27c3c0df645d4f1ca40e29c119e272d3]) successfully deleted
I20260926 13:09:07.138778 2720 tablet_bootstrap.cc:492] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9: No bootstrap required, opened a new log
I20260926 13:09:07.138825 2720 ts_tablet_manager.cc:1397] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:07.138967 2720 raft_consensus.cc:348] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.139040 2720 raft_consensus.cc:374] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:07.139066 2720 raft_consensus.cc:729] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d1a6afd64d42f8802cf1950d495fd9, State: Initialized, Role: FOLLOWER
I20260926 13:09:07.139106 2720 consensus_queue.cc:260] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.139181 2720 ts_tablet_manager.cc:1428] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:07.298659 2725 raft_consensus.cc:482] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:07.298931 2725 raft_consensus.cc:504] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.299358 2725 leader_election.cc:288] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2:45199), 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3:33543)
I20260926 13:09:07.303753 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:07.299686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39745 (local address 127.0.98.2:45199)
0926 13:09:07.299772 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:07.299774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:07.299810 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:09:07.299903 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:07.299904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:07.299916 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:07.299974 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:07.300274 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.300972 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.302548 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.303317 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.303403 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:07.303421 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:07.303596 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 13:09:07.303657 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:09:07.303810 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:07.299553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45199 (local address 127.0.98.1:39745)
0926 13:09:07.299722 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:07.299735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:07.299832 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:07.300045 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:07.300046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:07.300050 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:07.300161 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:07.300164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.301101 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.301103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:07.302408 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:07.302413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.303416 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.303417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:07.303498 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:07.303506 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 13:09:07.303572 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 13:09:07.303629 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":66,"thread_start_us":78,"threads_started":1}
I20260926 13:09:07.303920 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:07.299818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40921 (local address 127.0.98.3:33543)
0926 13:09:07.299989 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:07.299992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:07.300079 (+ 87us) server_negotiation.cc:407] Connection header received
0926 13:09:07.300160 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:07.300161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:07.300176 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:07.300257 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:07.300500 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.301082 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.302655 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.303316 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.303403 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:07.303420 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:07.303723 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 13:09:07.303829 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260926 13:09:07.303810 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:07.299756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33543 (local address 127.0.98.1:40921)
0926 13:09:07.299984 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:07.299997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:07.300072 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:07.300281 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:07.300283 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:07.300286 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:07.300396 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:07.300399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.301223 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.301224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:07.302499 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:07.302504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.303492 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.303493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:07.303583 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:07.303589 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:09:07.303653 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 13:09:07.303715 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":149,"threads_started":1}
I20260926 13:09:07.304217 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2704cc671b07462db7a38f0f984e1ace" candidate_uuid: "04abd56465fe4d14b2c1056ea7649d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" is_pre_election: true
I20260926 13:09:07.304343 2419 raft_consensus.cc:2454] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04abd56465fe4d14b2c1056ea7649d49 in term 0.
I20260926 13:09:07.304340 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2704cc671b07462db7a38f0f984e1ace" candidate_uuid: "04abd56465fe4d14b2c1056ea7649d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c1431a6e1614401bbf936a4f644d0cc" is_pre_election: true
I20260926 13:09:07.304435 2559 raft_consensus.cc:2454] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04abd56465fe4d14b2c1056ea7649d49 in term 0.
I20260926 13:09:07.304613 2213 leader_election.cc:302] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04abd56465fe4d14b2c1056ea7649d49, b4d1a6afd64d42f8802cf1950d495fd9; no voters:
I20260926 13:09:07.304800 2725 raft_consensus.cc:2781] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:07.304854 2725 raft_consensus.cc:482] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:07.304874 2725 raft_consensus.cc:3037] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:07.305668 2725 raft_consensus.cc:504] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.305859 2725 leader_election.cc:288] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [CANDIDATE]: Term 1 election: Requested vote from peers b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2:45199), 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3:33543)
I20260926 13:09:07.306149 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2704cc671b07462db7a38f0f984e1ace" candidate_uuid: "04abd56465fe4d14b2c1056ea7649d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4d1a6afd64d42f8802cf1950d495fd9"
I20260926 13:09:07.306159 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2704cc671b07462db7a38f0f984e1ace" candidate_uuid: "04abd56465fe4d14b2c1056ea7649d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c1431a6e1614401bbf936a4f644d0cc"
I20260926 13:09:07.306279 2559 raft_consensus.cc:3037] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:07.306279 2419 raft_consensus.cc:3037] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:07.307323 2559 raft_consensus.cc:2454] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04abd56465fe4d14b2c1056ea7649d49 in term 1.
I20260926 13:09:07.307323 2419 raft_consensus.cc:2454] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04abd56465fe4d14b2c1056ea7649d49 in term 1.
I20260926 13:09:07.307523 2213 leader_election.cc:302] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04abd56465fe4d14b2c1056ea7649d49, b4d1a6afd64d42f8802cf1950d495fd9; no voters:
I20260926 13:09:07.307637 2725 raft_consensus.cc:2781] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:07.307861 2725 raft_consensus.cc:686] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [term 1 LEADER]: Becoming Leader. State: Replica: 04abd56465fe4d14b2c1056ea7649d49, State: Running, Role: LEADER
I20260926 13:09:07.307956 2725 consensus_queue.cc:237] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } }
I20260926 13:09:07.308797 2122 catalog_manager.cc:5495] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "04abd56465fe4d14b2c1056ea7649d49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd56465fe4d14b2c1056ea7649d49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 } health_report { overall_health: UNKNOWN } } }
I20260926 13:09:07.327927 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:07.324982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.0.98.1:40577)
0926 13:09:07.325262 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:07.325264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:07.325272 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:07.325318 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:07.325319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:07.325343 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:07.325421 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:07.325625 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.326161 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.326960 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.327189 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.327267 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:07.327442 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:07.327566 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:07.327641 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:07.327740 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 13:09:07.327759 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":142,"threads_started":1}
I20260926 13:09:07.334115 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:07.331281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55277 (local address 127.0.98.3:33543)
0926 13:09:07.331336 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:07.331338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:07.331349 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:07.331403 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:07.331404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:07.331415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:07.331467 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:07.331862 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.332378 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.333358 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.333590 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.333662 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:07.333876 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:07.333909 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:07.333954 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:07.334014 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 13:09:07.334036 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:09:07.334141 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:07.331057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33543 (local address 127.0.98.1:55277)
0926 13:09:07.331267 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:07.331287 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:07.331342 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:07.331537 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:07.331538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:07.331541 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:07.331690 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:07.331693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.332566 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.332567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:07.333230 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:07.333232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.333701 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.333702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:07.333755 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:07.333776 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:07.333964 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:07.333965 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 13:09:07.334000 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 13:09:07.334045 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260926 13:09:07.340140 2559 raft_consensus.cc:1261] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Refusing update from remote peer 04abd56465fe4d14b2c1056ea7649d49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:07.340332 2419 raft_consensus.cc:1261] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Refusing update from remote peer 04abd56465fe4d14b2c1056ea7649d49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:07.340720 2738 consensus_queue.cc:1035] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4d1a6afd64d42f8802cf1950d495fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:07.340895 2725 consensus_queue.cc:1035] T 2704cc671b07462db7a38f0f984e1ace P 04abd56465fe4d14b2c1056ea7649d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c1431a6e1614401bbf936a4f644d0cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:07.350033 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:07.347325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40577 (local address 127.0.98.3:54161)
0926 13:09:07.347559 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:07.347571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:07.347637 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:07.347771 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:07.347771 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:07.347773 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:07.347821 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:07.347822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.348444 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.348444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:07.349186 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:07.349188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.349646 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.349647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:07.349706 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:07.349711 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:07.349855 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:07.349857 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:09:07.349901 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 13:09:07.349957 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":115,"threads_started":1}
I20260926 13:09:07.350019 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:07.347381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54161 (local address 127.0.98.1:40577)
0926 13:09:07.347432 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:07.347434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:07.347628 (+ 194us) server_negotiation.cc:407] Connection header received
0926 13:09:07.347707 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:07.347708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:07.347722 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:07.347771 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:07.347865 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.348380 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.349273 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:07.349577 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:07.349639 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:07.349770 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:07.349808 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:07.349850 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:07.349906 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 13:09:07.349935 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 13:09:07.350518 2690 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 13:09:08.876220 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:08.876348 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:08.877379 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f4d5984c7e741988487dfbc0ead00c7: Sending DeleteTablet for 3 replicas of tablet 2704cc671b07462db7a38f0f984e1ace
I20260926 13:09:08.877710 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 2704cc671b07462db7a38f0f984e1ace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47216
I20260926 13:09:08.877707 2242 tablet_service.cc:1514] Processing DeleteTablet for tablet 2704cc671b07462db7a38f0f984e1ace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49744
I20260926 13:09:08.877813 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 2704cc671b07462db7a38f0f984e1ace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60442
I20260926 13:09:08.878023 2775 tablet_replica.cc:330] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc: stopping tablet replica
I20260926 13:09:08.878046 2773 tablet_replica.cc:330] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9: stopping tablet replica
I20260926 13:09:08.878093 2775 raft_consensus.cc:2229] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:08.878106 2773 raft_consensus.cc:2229] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:08.878139 2775 raft_consensus.cc:2258] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:08.878144 2773 raft_consensus.cc:2258] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:08.878185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 2194
I20260926 13:09:08.879113 2773 ts_tablet_manager.cc:1905] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:08.880723 2775 ts_tablet_manager.cc:1905] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:08.880916 2773 ts_tablet_manager.cc:1918] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:09:08.880980 2773 log.cc:1196] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/2704cc671b07462db7a38f0f984e1ace
I20260926 13:09:08.881194 2773 ts_tablet_manager.cc:1939] T 2704cc671b07462db7a38f0f984e1ace P b4d1a6afd64d42f8802cf1950d495fd9: Deleting consensus metadata
I20260926 13:09:08.881587 2109 catalog_manager.cc:4841] TS b4d1a6afd64d42f8802cf1950d495fd9 (127.0.98.2:45199): tablet 2704cc671b07462db7a38f0f984e1ace (table test-table [id=6263170e0e16421b95c505c558e03c52]) successfully deleted
I20260926 13:09:08.882321 2775 ts_tablet_manager.cc:1918] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:09:08.882383 2775 log.cc:1196] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/2704cc671b07462db7a38f0f984e1ace
I20260926 13:09:08.882613 2775 ts_tablet_manager.cc:1939] T 2704cc671b07462db7a38f0f984e1ace P 1c1431a6e1614401bbf936a4f644d0cc: Deleting consensus metadata
I20260926 13:09:08.882977 2106 catalog_manager.cc:4841] TS 1c1431a6e1614401bbf936a4f644d0cc (127.0.98.3:33543): tablet 2704cc671b07462db7a38f0f984e1ace (table test-table [id=6263170e0e16421b95c505c558e03c52]) successfully deleted
W20260926 13:09:08.884841 2108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40577 (error 108)
I20260926 13:09:08.885190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 2334
I20260926 13:09:08.886123 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:08.885285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40577 (local address 127.0.0.1:34866)
0926 13:09:08.886041 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:08.886086 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":64,"threads_started":1}
W20260926 13:09:08.886216 2108 catalog_manager.cc:4584] TS 04abd56465fe4d14b2c1056ea7649d49 (127.0.98.1:40577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2704cc671b07462db7a38f0f984e1ace: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40577: connect: Connection refused (error 111)
I20260926 13:09:08.891098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 2474
I20260926 13:09:08.896332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 2090
2026-09-26T13:09:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5613 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:08 dist-test-slave-2dw5 krb5kdc[2780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:08 dist-test-slave-2dw5 krb5kdc[2780](info): set up 2 sockets
Sep 26 13:09:08 dist-test-slave-2dw5 krb5kdc[2780](info): commencing operation
krb5kdc: starting...
W20260926 13:09:09.862603 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:09 dist-test-slave-2dw5 krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:09:09.983162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34389
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:10.063512 2793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:10.063817 2793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:10.063853 2793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:10.065128 2793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:10.065178 2793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:10.065196 2793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:10.065213 2793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:10.065229 2793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:10.066905 2793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:10.067168 2793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:10.067380 2793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:10.072237 2799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:10.072240 2801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:10.072399 2798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:10.072636 2793 server_base.cc:1034] running on GCE node
I20260926 13:09:10.072918 2793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:10.073206 2793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:10.074371 2793 hybrid_clock.cc:648] HybridClock initialized: now 1790428150074353 us; error 37 us; skew 500 ppm
Sep 26 13:09:10 dist-test-slave-2dw5 krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:10.077044 2793 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:09:10.077514 2793 webserver.cc:460] Webserver started at http://127.0.98.62:33479/ using document root <none> and password file <none>
I20260926 13:09:10.077739 2793 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:10.077790 2793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:10.077900 2793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:10.079051 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "383f4a57c32d4858bb5e6a7c072ab436"
format_stamp: "Formatted at 2026-09-26 13:09:10 on dist-test-slave-2dw5"
server_key: "1219473e8bc5b2936bd1268d257c5011"
server_key_iv: "d598f2a5ef39777b47bb3bd03778ae40"
server_key_version: "encryptionkey@0"
I20260926 13:09:10.079398 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "383f4a57c32d4858bb5e6a7c072ab436"
format_stamp: "Formatted at 2026-09-26 13:09:10 on dist-test-slave-2dw5"
server_key: "1219473e8bc5b2936bd1268d257c5011"
server_key_iv: "d598f2a5ef39777b47bb3bd03778ae40"
server_key_version: "encryptionkey@0"
I20260926 13:09:10.080719 2793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:10.081532 2808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:10.081738 2793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:10.081825 2793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "383f4a57c32d4858bb5e6a7c072ab436"
format_stamp: "Formatted at 2026-09-26 13:09:10 on dist-test-slave-2dw5"
server_key: "1219473e8bc5b2936bd1268d257c5011"
server_key_iv: "d598f2a5ef39777b47bb3bd03778ae40"
server_key_version: "encryptionkey@0"
I20260926 13:09:10.081885 2793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:10.092808 2793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:10.094957 2793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:10.095068 2793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:10.098666 2793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34389
I20260926 13:09:10.098701 2860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34389 every 8 connection(s)
I20260926 13:09:10.099221 2793 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:10.099867 2861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:10.102108 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436: Bootstrap starting.
I20260926 13:09:10.102658 2861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:10.102949 2861 log.cc:824] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:10.103619 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436: No bootstrap required, opened a new log
I20260926 13:09:10.105175 2861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383f4a57c32d4858bb5e6a7c072ab436" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34389 } }
I20260926 13:09:10.105374 2861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:10.105399 2861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383f4a57c32d4858bb5e6a7c072ab436, State: Initialized, Role: FOLLOWER
I20260926 13:09:10.105516 2861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383f4a57c32d4858bb5e6a7c072ab436" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34389 } }
I20260926 13:09:10.105590 2861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:10.105618 2861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:10.105654 2861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:10.106537 2861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383f4a57c32d4858bb5e6a7c072ab436" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34389 } }
I20260926 13:09:10.106630 2861 leader_election.cc:302] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 383f4a57c32d4858bb5e6a7c072ab436; no voters:
I20260926 13:09:10.106772 2861 leader_election.cc:288] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:10.106875 2864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:10.107038 2861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:10.107048 2864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [term 1 LEADER]: Becoming Leader. State: Replica: 383f4a57c32d4858bb5e6a7c072ab436, State: Running, Role: LEADER
I20260926 13:09:10.107173 2864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383f4a57c32d4858bb5e6a7c072ab436" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34389 } }
I20260926 13:09:10.107640 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 383f4a57c32d4858bb5e6a7c072ab436. Latest consensus state: current_term: 1 leader_uuid: "383f4a57c32d4858bb5e6a7c072ab436" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383f4a57c32d4858bb5e6a7c072ab436" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34389 } } }
I20260926 13:09:10.107729 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:10.107888 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "383f4a57c32d4858bb5e6a7c072ab436" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383f4a57c32d4858bb5e6a7c072ab436" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34389 } } }
I20260926 13:09:10.108042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 2793
I20260926 13:09:10.108131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:10.108282 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:10.108306 392 external_mini_cluster.cc:1428] Setting key 38336d14a1ef98b941fb0ca70f567a3b
I20260926 13:09:10.108426 2873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:10.109361 2873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:10.111066 2873 catalog_manager.cc:1347] Generated new cluster ID: 470c699520ab4fa2961e8727f5977497
I20260926 13:09:10.111124 2873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:09:10 dist-test-slave-2dw5 krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:10.119726 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:10.108799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.0.98.62:34389)
0926 13:09:10.108994 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:10.109002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:10.109031 (+ 29us) server_negotiation.cc:407] Connection header received
0926 13:09:10.109135 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:10.109141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:10.109185 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:10.109323 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:10.109924 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.111153 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.112269 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.112905 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.113011 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:10.115914 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:10.115919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:10.115920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:10.115934 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:10.117994 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:10.118398 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:10.118401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:10.118402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:10.118525 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:10.118721 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:10.118722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:10.118722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:10.119086 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:10.119267 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:10.119340 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 13:09:10.119484 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":58,"threads_started":1}
I20260926 13:09:10.134266 2873 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:10.135102 2873 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:10.138238 2873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 383f4a57c32d4858bb5e6a7c072ab436: Generated new TSK 0
I20260926 13:09:10.138475 2873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:09:10.174189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34389
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:10.276496 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:10.276741 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:10.276765 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:10.278034 2887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:10.278086 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:10.278115 2887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:10.279865 2887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:10.280133 2887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:10.280364 2887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:10.285471 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:10.285488 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:10.285506 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:10.285528 2887 server_base.cc:1034] running on GCE node
I20260926 13:09:10.285884 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:10.286098 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:10.287343 2887 hybrid_clock.cc:648] HybridClock initialized: now 1790428150287320 us; error 30 us; skew 500 ppm
Sep 26 13:09:10 dist-test-slave-2dw5 krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:10.289968 2887 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:09:10.290460 2887 webserver.cc:460] Webserver started at http://127.0.98.1:39029/ using document root <none> and password file <none>
I20260926 13:09:10.290681 2887 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:10.290745 2887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:10.290916 2887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:10.292001 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "346266eb2d7349d9a5abf970cd06a3c3"
format_stamp: "Formatted at 2026-09-26 13:09:10 on dist-test-slave-2dw5"
server_key: "f245c34b93004653526c131f59b7bf74"
server_key_iv: "41294acb86f4ffcb1b709169fc9ea82a"
server_key_version: "encryptionkey@0"
I20260926 13:09:10.292344 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "346266eb2d7349d9a5abf970cd06a3c3"
format_stamp: "Formatted at 2026-09-26 13:09:10 on dist-test-slave-2dw5"
server_key: "f245c34b93004653526c131f59b7bf74"
server_key_iv: "41294acb86f4ffcb1b709169fc9ea82a"
server_key_version: "encryptionkey@0"
I20260926 13:09:10.293673 2887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:10.294584 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:10.294831 2887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:10.294955 2887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "346266eb2d7349d9a5abf970cd06a3c3"
format_stamp: "Formatted at 2026-09-26 13:09:10 on dist-test-slave-2dw5"
server_key: "f245c34b93004653526c131f59b7bf74"
server_key_iv: "41294acb86f4ffcb1b709169fc9ea82a"
server_key_version: "encryptionkey@0"
I20260926 13:09:10.295025 2887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:10.312253 2887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:10.314347 2887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:10.314482 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:10.314693 2887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:10.315042 2887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:10.315079 2887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:10.315109 2887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:10.315130 2887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:10.320724 2887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42001
I20260926 13:09:10.320809 3015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42001 every 8 connection(s)
I20260926 13:09:10.321089 2887 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:09:10 dist-test-slave-2dw5 krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:10.329950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 2887
I20260926 13:09:10.330049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:10.330138 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:10.321772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52911 (local address 127.0.98.62:34389)
0926 13:09:10.321833 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:10.321835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:10.322019 (+ 184us) server_negotiation.cc:407] Connection header received
0926 13:09:10.323030 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:10.323031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:10.323047 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:10.323110 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:10.323791 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.324304 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.325282 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.325483 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.325538 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:10.327668 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:10.327670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:10.327670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:10.327681 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:10.328796 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:10.329163 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:10.329164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:10.329164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:10.329218 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:10.329435 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:10.329435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:10.329436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:10.329550 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:10.329605 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:10.329957 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 13:09:10.330045 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:09:10.330235 392 external_mini_cluster.cc:1428] Setting key d86fe961b92a6c7978463935739d955e
I20260926 13:09:10.330305 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:10.321740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34389 (local address 127.0.98.1:52911)
0926 13:09:10.321907 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:10.321941 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:10.322925 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:10.323185 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:10.323189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:10.323463 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:10.323705 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:10.323710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.324383 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.324384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:10.325196 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:10.325198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.325550 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.325551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:10.325672 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:10.326157 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:10.326167 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:10.327569 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:10.328875 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:10.328882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:10.328885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:10.329098 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:10.329269 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:10.329270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:10.329270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:10.329377 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:10.329603 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:10.329610 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:10.329745 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 13:09:10.329908 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 13:09:10.330079 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260926 13:09:10.330788 3016 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34389
I20260926 13:09:10.330920 3016 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:10.331123 3016 heartbeater.cc:511] Master 127.0.98.62:34389 requested a full tablet report, sending...
I20260926 13:09:10.331676 2825 ts_manager.cc:194] Registered new tserver with Master: 346266eb2d7349d9a5abf970cd06a3c3 (127.0.98.1:42001)
I20260926 13:09:10.332455 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52911
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:09:10.354506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34389
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:10.437434 3021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:10.437687 3021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:10.437714 3021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:10.438952 3021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:10.439005 3021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:10.439036 3021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:10.440666 3021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:10.440905 3021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:10.441115 3021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:10.446444 3027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:10.446448 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:10.446444 3026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:10.446727 3021 server_base.cc:1034] running on GCE node
I20260926 13:09:10.447000 3021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:10.447225 3021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:10.448447 3021 hybrid_clock.cc:648] HybridClock initialized: now 1790428150448426 us; error 30 us; skew 500 ppm
Sep 26 13:09:10 dist-test-slave-2dw5 krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:10.451020 3021 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:09:10.451443 3021 webserver.cc:460] Webserver started at http://127.0.98.2:42681/ using document root <none> and password file <none>
I20260926 13:09:10.451660 3021 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:10.451704 3021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:10.451822 3021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:10.452878 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "8163eb175dc9434997f9e70240ee9523"
format_stamp: "Formatted at 2026-09-26 13:09:10 on dist-test-slave-2dw5"
server_key: "4cf7081ea07c1781d9d556636b65b7b7"
server_key_iv: "1e4aa688b2d00caa0abfb09cb1da44e4"
server_key_version: "encryptionkey@0"
I20260926 13:09:10.453202 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8163eb175dc9434997f9e70240ee9523"
format_stamp: "Formatted at 2026-09-26 13:09:10 on dist-test-slave-2dw5"
server_key: "4cf7081ea07c1781d9d556636b65b7b7"
server_key_iv: "1e4aa688b2d00caa0abfb09cb1da44e4"
server_key_version: "encryptionkey@0"
I20260926 13:09:10.454555 3021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:10.455430 3036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:10.455602 3021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:09:10.455686 3021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "8163eb175dc9434997f9e70240ee9523"
format_stamp: "Formatted at 2026-09-26 13:09:10 on dist-test-slave-2dw5"
server_key: "4cf7081ea07c1781d9d556636b65b7b7"
server_key_iv: "1e4aa688b2d00caa0abfb09cb1da44e4"
server_key_version: "encryptionkey@0"
I20260926 13:09:10.455796 3021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:10.471091 3021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:10.473233 3021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:10.473376 3021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:10.473599 3021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:10.473894 3021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:10.473925 3021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:10.473954 3021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:10.473978 3021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:10.479332 3021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43753
I20260926 13:09:10.479415 3149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43753 every 8 connection(s)
I20260926 13:09:10.479676 3021 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:09:10 dist-test-slave-2dw5 krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:10.489465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 3021
I20260926 13:09:10.489573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:10.489763 392 external_mini_cluster.cc:1428] Setting key 66dd22348a563dabf3ff7c49414f9d9d
I20260926 13:09:10.490022 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:10.480414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51825 (local address 127.0.98.62:34389)
0926 13:09:10.480463 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:10.480465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:10.480767 (+ 302us) server_negotiation.cc:407] Connection header received
0926 13:09:10.481715 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:10.481717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:10.481727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:10.481792 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:10.482455 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.483025 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.484087 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.484333 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.484384 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:10.486589 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:10.486590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:10.486591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:10.486599 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:10.488366 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:10.488911 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:10.488913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:10.488913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:10.488979 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:10.489242 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:10.489244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:10.489244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:10.489386 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:10.489455 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:10.489827 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 13:09:10.489918 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:09:10.490257 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:10.480365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34389 (local address 127.0.98.2:51825)
0926 13:09:10.480536 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:10.480614 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:10.481584 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:10.481891 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:10.481895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:10.482159 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:10.482353 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:10.482357 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.483117 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.483119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:10.483990 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:10.483993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.484412 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.484413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:10.484546 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:10.485040 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:10.485049 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:10.486495 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:10.488488 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:10.488497 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:10.488500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:10.488817 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:10.489042 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:10.489043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:10.489043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:10.489178 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:10.489461 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:10.489467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:10.489669 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 13:09:10.489795 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 13:09:10.490010 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260926 13:09:10.490777 3150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34389
I20260926 13:09:10.490898 3150 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:10.491099 3150 heartbeater.cc:511] Master 127.0.98.62:34389 requested a full tablet report, sending...
I20260926 13:09:10.491482 2825 ts_manager.cc:194] Registered new tserver with Master: 8163eb175dc9434997f9e70240ee9523 (127.0.98.2:43753)
I20260926 13:09:10.491850 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51825
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:09:10.513890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34389
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:10.595163 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:10.595383 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:10.595408 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:10.596568 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:10.596606 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:10.596635 3155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:10.598201 3155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:10.598407 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:10.598598 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:10.603560 3160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:10.603551 3163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:10.603562 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:10.603911 3155 server_base.cc:1034] running on GCE node
I20260926 13:09:10.604092 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:10.604321 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:10.605469 3155 hybrid_clock.cc:648] HybridClock initialized: now 1790428150605455 us; error 34 us; skew 500 ppm
Sep 26 13:09:10 dist-test-slave-2dw5 krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:10.608359 3155 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:09:10.608842 3155 webserver.cc:460] Webserver started at http://127.0.98.3:43341/ using document root <none> and password file <none>
I20260926 13:09:10.609043 3155 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:10.609090 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:10.609215 3155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:10.610297 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "441aaa4e59694a08980ce51505668d5f"
format_stamp: "Formatted at 2026-09-26 13:09:10 on dist-test-slave-2dw5"
server_key: "d0a12c1435136163fb998f257decc065"
server_key_iv: "54f8e0f788e79fb649e18313d420a4c7"
server_key_version: "encryptionkey@0"
I20260926 13:09:10.610633 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "441aaa4e59694a08980ce51505668d5f"
format_stamp: "Formatted at 2026-09-26 13:09:10 on dist-test-slave-2dw5"
server_key: "d0a12c1435136163fb998f257decc065"
server_key_iv: "54f8e0f788e79fb649e18313d420a4c7"
server_key_version: "encryptionkey@0"
I20260926 13:09:10.611963 3155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:09:10.612778 3170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:10.612965 3155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:10.613047 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "441aaa4e59694a08980ce51505668d5f"
format_stamp: "Formatted at 2026-09-26 13:09:10 on dist-test-slave-2dw5"
server_key: "d0a12c1435136163fb998f257decc065"
server_key_iv: "54f8e0f788e79fb649e18313d420a4c7"
server_key_version: "encryptionkey@0"
I20260926 13:09:10.613116 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:10.625813 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:10.627923 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:10.628068 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:10.628291 3155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:10.628575 3155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:10.628610 3155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:10.628640 3155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:10.628667 3155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:10.634718 3155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46837
I20260926 13:09:10.634791 3283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46837 every 8 connection(s)
I20260926 13:09:10.635154 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:10.638729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 3155
I20260926 13:09:10.638823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:10.639014 392 external_mini_cluster.cc:1428] Setting key fa8b063e1f394b49d1b3a50f57c6ea4f
Sep 26 13:09:10 dist-test-slave-2dw5 krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:10.645205 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:10.635906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45965 (local address 127.0.98.62:34389)
0926 13:09:10.635951 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:10.635953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:10.636167 (+ 214us) server_negotiation.cc:407] Connection header received
0926 13:09:10.637135 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:10.637136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:10.637151 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:10.637217 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:10.637873 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.638399 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.639734 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.639988 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.640047 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:10.642316 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:10.642318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:10.642318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:10.642325 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:10.643860 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:10.644379 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:10.644380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:10.644380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:10.644433 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:10.644683 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:10.644683 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:10.644683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:10.644785 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:10.644862 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:10.645072 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 13:09:10.645136 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:09:10.645501 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:10.635882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34389 (local address 127.0.98.3:45965)
0926 13:09:10.636033 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:10.636071 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:10.637011 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:10.637294 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:10.637298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:10.637572 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:10.637775 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:10.637779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.638494 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.638496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:10.639651 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:10.639654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:10.640072 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:10.640072 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:10.640195 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:10.640716 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:10.640725 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:10.642113 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:10.643952 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:10.643960 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:10.643961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:10.644305 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:10.644503 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:10.644504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:10.644504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:10.644627 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:10.644828 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:10.644831 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:10.644953 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 13:09:10.645043 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 13:09:10.645279 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260926 13:09:10.645927 3284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34389
I20260926 13:09:10.646019 3284 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:10.646202 3284 heartbeater.cc:511] Master 127.0.98.62:34389 requested a full tablet report, sending...
I20260926 13:09:10.646560 2825 ts_manager.cc:194] Registered new tserver with Master: 441aaa4e59694a08980ce51505668d5f (127.0.98.3:46837)
I20260926 13:09:10.647001 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45965
I20260926 13:09:10.650894 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 13:09:10.661901 2882 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:10.660937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.0.98.62:34389)
0926 13:09:10.660993 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:10.660995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:10.661137 (+ 142us) server_negotiation.cc:407] Connection header received
0926 13:09:10.661169 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:10.661170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:10.661181 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:10.661248 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:10.661820 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43874: BlockingRecv error: recv got EOF from 127.0.0.1:43874 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:09:10.663087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 2887
I20260926 13:09:10.667136 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 3021
I20260926 13:09:10.671085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 3155
I20260926 13:09:10.674953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 2793
2026-09-26T13:09:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1777 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:10 dist-test-slave-2dw5 krb5kdc[3296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:10 dist-test-slave-2dw5 krb5kdc[3296](info): set up 2 sockets
Sep 26 13:09:10 dist-test-slave-2dw5 krb5kdc[3296](info): commencing operation
krb5kdc: starting...
W20260926 13:09:11.634545 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:11 dist-test-slave-2dw5 krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:09:11.754767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44021
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:11.836376 3309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:11.836604 3309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:11.836627 3309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:11.837915 3309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:11.837973 3309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:11.837986 3309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:11.837998 3309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:11.838009 3309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:11.839795 3309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:11.840055 3309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:11.840282 3309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:11.845084 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:11.845091 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:11.845140 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:11.845283 3309 server_base.cc:1034] running on GCE node
I20260926 13:09:11.845530 3309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:11.845815 3309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:11.846977 3309 hybrid_clock.cc:648] HybridClock initialized: now 1790428151846954 us; error 31 us; skew 500 ppm
Sep 26 13:09:11 dist-test-slave-2dw5 krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:11.849359 3309 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:09:11.849835 3309 webserver.cc:460] Webserver started at http://127.0.98.62:32927/ using document root <none> and password file <none>
I20260926 13:09:11.850056 3309 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:11.850102 3309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:11.850225 3309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:11.851318 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "964051e0cba14c9c993c996ddd502d54"
format_stamp: "Formatted at 2026-09-26 13:09:11 on dist-test-slave-2dw5"
server_key: "618814a3fd09a722cb2e31d00576aa80"
server_key_iv: "390f15485873b45855161451f9f500c1"
server_key_version: "encryptionkey@0"
I20260926 13:09:11.851634 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "964051e0cba14c9c993c996ddd502d54"
format_stamp: "Formatted at 2026-09-26 13:09:11 on dist-test-slave-2dw5"
server_key: "618814a3fd09a722cb2e31d00576aa80"
server_key_iv: "390f15485873b45855161451f9f500c1"
server_key_version: "encryptionkey@0"
I20260926 13:09:11.852842 3309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:11.853497 3324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:11.853643 3309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:11.853724 3309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "964051e0cba14c9c993c996ddd502d54"
format_stamp: "Formatted at 2026-09-26 13:09:11 on dist-test-slave-2dw5"
server_key: "618814a3fd09a722cb2e31d00576aa80"
server_key_iv: "390f15485873b45855161451f9f500c1"
server_key_version: "encryptionkey@0"
I20260926 13:09:11.853788 3309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:11.872577 3309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:11.874668 3309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:11.874816 3309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:11.878630 3309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44021
I20260926 13:09:11.878688 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44021 every 8 connection(s)
I20260926 13:09:11.879041 3309 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:11.879475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 3309
I20260926 13:09:11.879555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:11.879721 392 external_mini_cluster.cc:1428] Setting key 4ba23e89d7238d08e1041bfa2f5c80aa
I20260926 13:09:11.880033 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:11.883407 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54: Bootstrap starting.
I20260926 13:09:11.884080 3377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:11.884392 3377 log.cc:824] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:11.885164 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54: No bootstrap required, opened a new log
Sep 26 13:09:11 dist-test-slave-2dw5 krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:11.887037 3377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964051e0cba14c9c993c996ddd502d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44021 } }
I20260926 13:09:11.887187 3377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:11.887219 3377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964051e0cba14c9c993c996ddd502d54, State: Initialized, Role: FOLLOWER
I20260926 13:09:11.887348 3377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964051e0cba14c9c993c996ddd502d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44021 } }
I20260926 13:09:11.887419 3377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:11.887450 3377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:11.887498 3377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:11.888046 3377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964051e0cba14c9c993c996ddd502d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44021 } }
I20260926 13:09:11.888140 3377 leader_election.cc:302] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 964051e0cba14c9c993c996ddd502d54; no voters:
I20260926 13:09:11.888311 3377 leader_election.cc:288] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:11.888370 3382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:11.888521 3382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [term 1 LEADER]: Becoming Leader. State: Replica: 964051e0cba14c9c993c996ddd502d54, State: Running, Role: LEADER
I20260926 13:09:11.888569 3377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:11.888686 3382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964051e0cba14c9c993c996ddd502d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44021 } }
I20260926 13:09:11.889071 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "964051e0cba14c9c993c996ddd502d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964051e0cba14c9c993c996ddd502d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44021 } } }
I20260926 13:09:11.889149 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:11.889357 3387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:11.889796 3387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:11.890527 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:11.880185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.0.98.62:44021)
0926 13:09:11.880380 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:11.880389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:11.880424 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:09:11.880909 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:11.880919 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:11.880983 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:11.881137 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:11.881807 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:11.883177 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:11.884052 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:11.884647 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:11.884761 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:11.886944 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:11.886949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:11.886949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:11.886963 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:11.888794 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:11.889229 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:11.889233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:11.889233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:11.889344 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:11.889619 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:11.889620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:11.889620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:11.889957 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:11.890125 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:11.890205 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 13:09:11.890341 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260926 13:09:11.891136 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 964051e0cba14c9c993c996ddd502d54. Latest consensus state: current_term: 1 leader_uuid: "964051e0cba14c9c993c996ddd502d54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964051e0cba14c9c993c996ddd502d54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44021 } } }
I20260926 13:09:11.891227 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:11.891636 3387 catalog_manager.cc:1347] Generated new cluster ID: 781e889c1a024109bbda9f817f02b38c
I20260926 13:09:11.891685 3387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:11.891693 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:09:11.908952 3387 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:11.909677 3387 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:11.911720 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:11.908184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.0.98.62:44021)
0926 13:09:11.908248 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:11.908250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:11.908263 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:09:11.908333 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:11.908334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:11.908345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:11.908403 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:11.908984 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:11.909582 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:11.910331 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:11.910546 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:11.910603 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:11.911286 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:11.911288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:11.911288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:11.911469 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:11.911486 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:11.911491 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:11.911541 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:11.911604 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 13:09:11.911645 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:09:11.920177 3387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 964051e0cba14c9c993c996ddd502d54: Generated new TSK 0
I20260926 13:09:11.920395 3387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:09:11.924304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 3309
2026-09-26T13:09:11Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1247 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:11 dist-test-slave-2dw5 krb5kdc[3410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:11 dist-test-slave-2dw5 krb5kdc[3410](info): set up 2 sockets
Sep 26 13:09:11 dist-test-slave-2dw5 krb5kdc[3410](info): commencing operation
krb5kdc: starting...
W20260926 13:09:12.878571 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:12 dist-test-slave-2dw5 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:09:12.997891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33795
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:13.080651 3423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:13.080878 3423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:13.080900 3423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:13.082028 3423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:13.082075 3423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:13.082088 3423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:13.082103 3423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:13.082113 3423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:13.084115 3423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:13.084340 3423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:13.084519 3423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:13.089232 3431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:13.089277 3429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:13.089303 3428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:13.089473 3423 server_base.cc:1034] running on GCE node
I20260926 13:09:13.089773 3423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:13.090090 3423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:13.091302 3423 hybrid_clock.cc:648] HybridClock initialized: now 1790428153091287 us; error 28 us; skew 500 ppm
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:13.093878 3423 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:09:13.094318 3423 webserver.cc:460] Webserver started at http://127.0.98.62:36913/ using document root <none> and password file <none>
I20260926 13:09:13.094537 3423 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:13.094576 3423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:13.094668 3423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:13.095783 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "770d152e9b2c4f589c6253d590f5c3f7"
format_stamp: "Formatted at 2026-09-26 13:09:13 on dist-test-slave-2dw5"
server_key: "4599eac9a2d3ef8ff4d869430c0fa073"
server_key_iv: "4625574f4bf38dfef54c39e0a4d4c369"
server_key_version: "encryptionkey@0"
I20260926 13:09:13.096083 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "770d152e9b2c4f589c6253d590f5c3f7"
format_stamp: "Formatted at 2026-09-26 13:09:13 on dist-test-slave-2dw5"
server_key: "4599eac9a2d3ef8ff4d869430c0fa073"
server_key_iv: "4625574f4bf38dfef54c39e0a4d4c369"
server_key_version: "encryptionkey@0"
I20260926 13:09:13.097296 3423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:09:13.097993 3438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:13.098174 3423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:13.098253 3423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "770d152e9b2c4f589c6253d590f5c3f7"
format_stamp: "Formatted at 2026-09-26 13:09:13 on dist-test-slave-2dw5"
server_key: "4599eac9a2d3ef8ff4d869430c0fa073"
server_key_iv: "4625574f4bf38dfef54c39e0a4d4c369"
server_key_version: "encryptionkey@0"
I20260926 13:09:13.098318 3423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:13.116551 3423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:13.118706 3423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:13.118891 3423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:13.122723 3423 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33795
I20260926 13:09:13.122767 3490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33795 every 8 connection(s)
I20260926 13:09:13.123172 3423 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:13.124182 3491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:13.126516 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7: Bootstrap starting.
I20260926 13:09:13.127118 3491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:13.127365 3491 log.cc:824] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:13.128096 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7: No bootstrap required, opened a new log
I20260926 13:09:13.129851 3491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770d152e9b2c4f589c6253d590f5c3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } }
I20260926 13:09:13.129998 3491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:13.130023 3491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770d152e9b2c4f589c6253d590f5c3f7, State: Initialized, Role: FOLLOWER
I20260926 13:09:13.130159 3491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770d152e9b2c4f589c6253d590f5c3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } }
I20260926 13:09:13.130218 3491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:13.130249 3491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:13.130293 3491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:13.131135 3491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770d152e9b2c4f589c6253d590f5c3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } }
I20260926 13:09:13.131227 3491 leader_election.cc:302] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 770d152e9b2c4f589c6253d590f5c3f7; no voters:
I20260926 13:09:13.131387 3491 leader_election.cc:288] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:13.131426 3494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:13.131556 3494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [term 1 LEADER]: Becoming Leader. State: Replica: 770d152e9b2c4f589c6253d590f5c3f7, State: Running, Role: LEADER
I20260926 13:09:13.131641 3491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:13.131707 3494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770d152e9b2c4f589c6253d590f5c3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } }
I20260926 13:09:13.132057 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 770d152e9b2c4f589c6253d590f5c3f7. Latest consensus state: current_term: 1 leader_uuid: "770d152e9b2c4f589c6253d590f5c3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770d152e9b2c4f589c6253d590f5c3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } }
I20260926 13:09:13.132053 3495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "770d152e9b2c4f589c6253d590f5c3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770d152e9b2c4f589c6253d590f5c3f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } }
I20260926 13:09:13.132222 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:13.132234 3495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:13.132452 3503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:13.133049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 3423
I20260926 13:09:13.133070 3503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:13.133143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:13.133327 392 external_mini_cluster.cc:1428] Setting key 6fb3c0e388f9c5a5def2436926258a59
I20260926 13:09:13.135219 3503 catalog_manager.cc:1347] Generated new cluster ID: 67abae81874b444b97ae718906ba07ea
I20260926 13:09:13.135283 3503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:13.143087 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:13.134046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.0.98.62:33795)
0926 13:09:13.134196 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:13.134205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:13.134236 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:09:13.134364 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:13.134372 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:13.134440 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:13.134577 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:13.135363 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.136375 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.137109 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.137596 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.137697 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:13.140041 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:13.140045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:13.140045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:13.140059 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:13.141631 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:13.141949 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:13.141952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:13.141952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:13.142068 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:13.142236 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:13.142236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:13.142237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:13.142486 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:13.142637 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:13.142708 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 13:09:13.142874 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260926 13:09:13.144151 3503 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:13.144834 3503 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:13.154201 3503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 770d152e9b2c4f589c6253d590f5c3f7: Generated new TSK 0
I20260926 13:09:13.154407 3503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:09:13.181321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33795
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:13.262753 3517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:13.263008 3517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:13.263036 3517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:13.264278 3517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:13.264324 3517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:13.264353 3517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:13.265873 3517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:13.266136 3517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:13.266371 3517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:13.271409 3523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:13.271409 3525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:13.271404 3522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:13.271472 3517 server_base.cc:1034] running on GCE node
I20260926 13:09:13.271786 3517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:13.272015 3517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:13.273200 3517 hybrid_clock.cc:648] HybridClock initialized: now 1790428153273182 us; error 27 us; skew 500 ppm
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:13.280100 3517 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:09:13.280615 3517 webserver.cc:460] Webserver started at http://127.0.98.1:35043/ using document root <none> and password file <none>
I20260926 13:09:13.280841 3517 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:13.280890 3517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:13.281057 3517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:13.282109 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "7cae443652e6496ebd1e4e78dbf20812"
format_stamp: "Formatted at 2026-09-26 13:09:13 on dist-test-slave-2dw5"
server_key: "d661ccdb6e688921e5c26cf67aba40a5"
server_key_iv: "0efb125e273521ada91f54be2a48aa6b"
server_key_version: "encryptionkey@0"
I20260926 13:09:13.282464 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7cae443652e6496ebd1e4e78dbf20812"
format_stamp: "Formatted at 2026-09-26 13:09:13 on dist-test-slave-2dw5"
server_key: "d661ccdb6e688921e5c26cf67aba40a5"
server_key_iv: "0efb125e273521ada91f54be2a48aa6b"
server_key_version: "encryptionkey@0"
I20260926 13:09:13.283843 3517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:13.284761 3532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:13.284937 3517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:13.285027 3517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "7cae443652e6496ebd1e4e78dbf20812"
format_stamp: "Formatted at 2026-09-26 13:09:13 on dist-test-slave-2dw5"
server_key: "d661ccdb6e688921e5c26cf67aba40a5"
server_key_iv: "0efb125e273521ada91f54be2a48aa6b"
server_key_version: "encryptionkey@0"
I20260926 13:09:13.285089 3517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:13.295676 3517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:13.297745 3517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:13.297878 3517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:13.298079 3517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:13.298375 3517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:13.298409 3517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:13.298439 3517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:13.298463 3517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:13.304236 3517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40151
I20260926 13:09:13.304334 3645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40151 every 8 connection(s)
I20260926 13:09:13.304611 3517 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:09:13.306392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 3517
I20260926 13:09:13.306497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:13.306720 392 external_mini_cluster.cc:1428] Setting key fc4be6f14442a30bcfe846dc50906a8f
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:09:13.314940 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:13.305339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54337 (local address 127.0.98.62:33795)
0926 13:09:13.305390 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:13.305393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:13.305603 (+ 210us) server_negotiation.cc:407] Connection header received
0926 13:09:13.306173 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:13.306174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:13.306185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:13.306239 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:13.307403 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.307903 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.309161 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.309447 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.309519 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:13.311639 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:13.311641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:13.311641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:13.311649 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:13.313324 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:13.313854 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:13.313856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:13.313856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:13.313916 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:13.314221 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:13.314223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:13.314223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:13.314367 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:13.314433 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:13.314731 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 13:09:13.314816 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:09:13.315069 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:13.305318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33795 (local address 127.0.98.1:54337)
0926 13:09:13.305481 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:13.305518 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:13.306015 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:13.306329 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:13.306334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:13.307107 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:13.307322 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:13.307329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.307990 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.307992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:13.309068 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:13.309071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.309538 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.309539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:13.309660 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:13.310168 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:13.310176 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:13.311545 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:13.313517 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:13.313525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:13.313528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:13.313773 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:13.314015 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:13.314016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:13.314016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:13.314148 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:13.314431 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:13.314434 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:13.314586 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 13:09:13.314691 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 13:09:13.314834 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260926 13:09:13.315682 3646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33795
I20260926 13:09:13.315778 3646 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:13.315956 3646 heartbeater.cc:511] Master 127.0.98.62:33795 requested a full tablet report, sending...
I20260926 13:09:13.316493 3455 ts_manager.cc:194] Registered new tserver with Master: 7cae443652e6496ebd1e4e78dbf20812 (127.0.98.1:40151)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 13:09:13.317314 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:09:13.333853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33795
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:13.415417 3651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:13.415652 3651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:13.415679 3651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:13.416941 3651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:13.416990 3651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:13.417019 3651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:13.418509 3651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:13.418737 3651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:13.418974 3651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:13.423959 3659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:13.424022 3656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:13.424153 3651 server_base.cc:1034] running on GCE node
W20260926 13:09:13.424041 3657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:13.424389 3651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:13.424599 3651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:13.425758 3651 hybrid_clock.cc:648] HybridClock initialized: now 1790428153425741 us; error 36 us; skew 500 ppm
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:13.428340 3651 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:09:13.428805 3651 webserver.cc:460] Webserver started at http://127.0.98.2:38109/ using document root <none> and password file <none>
I20260926 13:09:13.429003 3651 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:13.429050 3651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:13.429178 3651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:13.430260 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "4730d36c919d4bb5bd1a342b4b6058bb"
format_stamp: "Formatted at 2026-09-26 13:09:13 on dist-test-slave-2dw5"
server_key: "c5660d127abd7c4ebcc7f8d124122ec3"
server_key_iv: "7bf2fb5aab668112616b258d8d649d3e"
server_key_version: "encryptionkey@0"
I20260926 13:09:13.430608 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4730d36c919d4bb5bd1a342b4b6058bb"
format_stamp: "Formatted at 2026-09-26 13:09:13 on dist-test-slave-2dw5"
server_key: "c5660d127abd7c4ebcc7f8d124122ec3"
server_key_iv: "7bf2fb5aab668112616b258d8d649d3e"
server_key_version: "encryptionkey@0"
I20260926 13:09:13.431865 3651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:09:13.432559 3666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:13.432772 3651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:13.432853 3651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "4730d36c919d4bb5bd1a342b4b6058bb"
format_stamp: "Formatted at 2026-09-26 13:09:13 on dist-test-slave-2dw5"
server_key: "c5660d127abd7c4ebcc7f8d124122ec3"
server_key_iv: "7bf2fb5aab668112616b258d8d649d3e"
server_key_version: "encryptionkey@0"
I20260926 13:09:13.432926 3651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:13.443527 3651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:13.445518 3651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:13.445660 3651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:13.445876 3651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:13.446156 3651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:13.446188 3651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:13.446216 3651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:13.446234 3651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:13.451808 3651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42973
I20260926 13:09:13.451926 3779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42973 every 8 connection(s)
I20260926 13:09:13.452167 3651 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:13.459038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 3651
I20260926 13:09:13.459175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:13.459354 392 external_mini_cluster.cc:1428] Setting key ef4c27385097566496edd2fb0e3804e9
I20260926 13:09:13.460807 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:13.452842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60365 (local address 127.0.98.62:33795)
0926 13:09:13.452891 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:13.452893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:13.453072 (+ 179us) server_negotiation.cc:407] Connection header received
0926 13:09:13.453540 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:13.453541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:13.453551 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:13.453597 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:13.454478 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.455031 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.455977 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.456193 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.456242 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:13.458161 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:13.458163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:13.458163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:13.458169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:13.459454 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:13.459892 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:13.459893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:13.459894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:13.459953 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:13.460197 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:13.460198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:13.460198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:13.460293 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:13.460339 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:13.460682 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 13:09:13.460744 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:09:13.461051 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:13.452828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33795 (local address 127.0.98.2:60365)
0926 13:09:13.452966 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:13.453003 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:13.453464 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:13.453651 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:13.453654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:13.454213 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:13.454399 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:13.454403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.455098 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.455098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:13.455915 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:13.455917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.456247 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.456247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:13.456337 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:13.456782 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:13.456790 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:13.458077 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:13.459521 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:13.459527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:13.459529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:13.459826 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:13.460006 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:13.460006 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:13.460007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:13.460136 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:13.460343 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:13.460349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:13.460543 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 13:09:13.460672 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 13:09:13.460835 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260926 13:09:13.461565 3780 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33795
I20260926 13:09:13.461681 3780 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:13.461867 3780 heartbeater.cc:511] Master 127.0.98.62:33795 requested a full tablet report, sending...
I20260926 13:09:13.462266 3455 ts_manager.cc:194] Registered new tserver with Master: 4730d36c919d4bb5bd1a342b4b6058bb (127.0.98.2:42973)
I20260926 13:09:13.462774 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60365
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:09:13.485144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33795
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:13.566694 3785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:13.566958 3785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:13.566987 3785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:13.568200 3785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:13.568260 3785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:13.568289 3785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:13.569809 3785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:13.570056 3785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:13.570287 3785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:13.575418 3791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:13.575477 3785 server_base.cc:1034] running on GCE node
W20260926 13:09:13.575423 3790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:13.575464 3793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:13.575870 3785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:13.576094 3785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:13.577244 3785 hybrid_clock.cc:648] HybridClock initialized: now 1790428153577229 us; error 31 us; skew 500 ppm
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:13.579953 3785 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:09:13.580381 3785 webserver.cc:460] Webserver started at http://127.0.98.3:38675/ using document root <none> and password file <none>
I20260926 13:09:13.580602 3785 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:13.580646 3785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:13.580740 3785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:13.581728 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "16f771d8d262442c9b5aea07b616c5b6"
format_stamp: "Formatted at 2026-09-26 13:09:13 on dist-test-slave-2dw5"
server_key: "a256eb3e8242c005711caf9b1c2c19c2"
server_key_iv: "bf489ba0fceda67714692269893b5466"
server_key_version: "encryptionkey@0"
I20260926 13:09:13.582010 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "16f771d8d262442c9b5aea07b616c5b6"
format_stamp: "Formatted at 2026-09-26 13:09:13 on dist-test-slave-2dw5"
server_key: "a256eb3e8242c005711caf9b1c2c19c2"
server_key_iv: "bf489ba0fceda67714692269893b5466"
server_key_version: "encryptionkey@0"
I20260926 13:09:13.583290 3785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:13.584152 3800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:13.584352 3785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:13.584434 3785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "16f771d8d262442c9b5aea07b616c5b6"
format_stamp: "Formatted at 2026-09-26 13:09:13 on dist-test-slave-2dw5"
server_key: "a256eb3e8242c005711caf9b1c2c19c2"
server_key_iv: "bf489ba0fceda67714692269893b5466"
server_key_version: "encryptionkey@0"
I20260926 13:09:13.584491 3785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:13.593780 3785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:13.595832 3785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:13.595942 3785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:13.596140 3785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:13.596413 3785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:13.596446 3785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:13.596468 3785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:13.596493 3785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:13.602109 3785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35719
I20260926 13:09:13.602172 3913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35719 every 8 connection(s)
I20260926 13:09:13.602479 3785 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:13.610224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 3785
I20260926 13:09:13.610337 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:13.610505 392 external_mini_cluster.cc:1428] Setting key 887cc114a868ea2f5b3685b1360633e8
I20260926 13:09:13.611776 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:13.603149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40673 (local address 127.0.98.62:33795)
0926 13:09:13.603207 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:13.603209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:13.603421 (+ 212us) server_negotiation.cc:407] Connection header received
0926 13:09:13.603922 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:13.603924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:13.603936 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:13.603995 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:13.604916 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.605412 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.606379 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.606563 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.606615 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:13.608973 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:13.608974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:13.608975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:13.608982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:13.610307 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:13.610818 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:13.610819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:13.610820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:13.610913 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:13.611165 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:13.611166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:13.611166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:13.611293 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:13.611347 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:13.611606 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 13:09:13.611685 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:09:13.611992 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:13.603143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33795 (local address 127.0.98.3:40673)
0926 13:09:13.603301 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:13.603338 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:13.603831 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:13.604069 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:13.604074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:13.604636 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:13.604822 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:13.604827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.605490 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.605491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:13.606299 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:13.606302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.606631 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.606631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:13.606758 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:13.607310 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:13.607318 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:13.608886 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:13.610465 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:13.610472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:13.610474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:13.610739 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:13.610969 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:13.610970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:13.610970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:13.611104 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:13.611347 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:13.611353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:13.611488 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 13:09:13.611578 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 13:09:13.611760 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260926 13:09:13.612425 3914 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33795
I20260926 13:09:13.612593 3914 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:13.612771 3914 heartbeater.cc:511] Master 127.0.98.62:33795 requested a full tablet report, sending...
I20260926 13:09:13.613238 3455 ts_manager.cc:194] Registered new tserver with Master: 16f771d8d262442c9b5aea07b616c5b6 (127.0.98.3:35719)
I20260926 13:09:13.613592 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40673
I20260926 13:09:13.622313 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428153, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428153, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:13.644510 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:13.637625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.0.98.62:33795)
0926 13:09:13.637681 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:13.637683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:13.637726 (+ 43us) server_negotiation.cc:407] Connection header received
0926 13:09:13.637786 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:13.637787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:13.637796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:13.637841 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:13.638385 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.638938 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.639814 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:13.639998 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:13.640044 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:13.642226 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:13.642227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:13.642228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:13.642234 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:13.643533 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:13.643829 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:13.643830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:13.643830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:13.643878 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:13.644085 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:13.644085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:13.644086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:13.644231 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:13.644297 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:13.644385 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 13:09:13.644442 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 13:09:13.644783 3455 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34604
I20260926 13:09:13.645217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 3517
I20260926 13:09:13.649020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 3651
I20260926 13:09:13.653076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 3785
I20260926 13:09:13.656800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 3423
2026-09-26T13:09:13Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1733 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3926](info): set up 2 sockets
Sep 26 13:09:13 dist-test-slave-2dw5 krb5kdc[3926](info): commencing operation
krb5kdc: starting...
W20260926 13:09:14.610595 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:14 dist-test-slave-2dw5 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:09:14.725345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41627
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:14.810530 3939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:14.810738 3939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:14.810760 3939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:14.811959 3939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:14.812006 3939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:14.812018 3939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:14.812029 3939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:14.812040 3939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:14.813511 3939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:14.813719 3939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:14.813900 3939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:14.818605 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:14.818655 3944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:14.818655 3945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:14.819036 3939 server_base.cc:1034] running on GCE node
I20260926 13:09:14.819319 3939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:14.819630 3939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:14.820801 3939 hybrid_clock.cc:648] HybridClock initialized: now 1790428154820786 us; error 27 us; skew 500 ppm
Sep 26 13:09:14 dist-test-slave-2dw5 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:14.823622 3939 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:09:14.824056 3939 webserver.cc:460] Webserver started at http://127.0.98.62:33607/ using document root <none> and password file <none>
I20260926 13:09:14.824235 3939 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:14.824275 3939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:14.824362 3939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:14.825433 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "ce3709e2416345789234ec12bc6001be"
format_stamp: "Formatted at 2026-09-26 13:09:14 on dist-test-slave-2dw5"
server_key: "bc556d2877a8256cb95fcf1f7cb491b9"
server_key_iv: "b2a6ca3771bdceb2d58227c5069d3d62"
server_key_version: "encryptionkey@0"
I20260926 13:09:14.825783 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce3709e2416345789234ec12bc6001be"
format_stamp: "Formatted at 2026-09-26 13:09:14 on dist-test-slave-2dw5"
server_key: "bc556d2877a8256cb95fcf1f7cb491b9"
server_key_iv: "b2a6ca3771bdceb2d58227c5069d3d62"
server_key_version: "encryptionkey@0"
I20260926 13:09:14.827354 3939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 13:09:14.828254 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:14.828464 3939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:14.828552 3939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "ce3709e2416345789234ec12bc6001be"
format_stamp: "Formatted at 2026-09-26 13:09:14 on dist-test-slave-2dw5"
server_key: "bc556d2877a8256cb95fcf1f7cb491b9"
server_key_iv: "b2a6ca3771bdceb2d58227c5069d3d62"
server_key_version: "encryptionkey@0"
I20260926 13:09:14.828611 3939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:14.861687 3939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:14.863982 3939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:14.864110 3939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:14.868415 3939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41627
I20260926 13:09:14.868460 4006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41627 every 8 connection(s)
I20260926 13:09:14.868775 3939 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:14.869737 4007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:14.870612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 3939
I20260926 13:09:14.870729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:14.870919 392 external_mini_cluster.cc:1428] Setting key 967f47025d820f469375e535569ebb93
I20260926 13:09:14.873322 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be: Bootstrap starting.
I20260926 13:09:14.873914 4007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:14.874191 4007 log.cc:824] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:14.875080 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be: No bootstrap required, opened a new log
I20260926 13:09:14.877451 4007 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce3709e2416345789234ec12bc6001be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41627 } }
I20260926 13:09:14.877605 4007 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:14.877640 4007 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce3709e2416345789234ec12bc6001be, State: Initialized, Role: FOLLOWER
I20260926 13:09:14.877843 4007 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce3709e2416345789234ec12bc6001be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41627 } }
Sep 26 13:09:14 dist-test-slave-2dw5 krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:14.877907 4007 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:14.877929 4007 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:14.877975 4007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:14.881383 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:14.871474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.0.98.62:41627)
0926 13:09:14.871632 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:14.871640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:14.871666 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:09:14.872149 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:14.872156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:14.872208 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:14.872332 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:14.873236 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:14.874213 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:14.875193 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:14.875680 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:14.875764 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:14.878381 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:14.878386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:14.878386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:14.878399 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:14.879943 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:14.880287 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:14.880290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:14.880290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:14.880376 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:14.880560 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:14.880561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:14.880562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:14.880809 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:14.880942 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:14.881061 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 13:09:14.881195 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260926 13:09:14.907312 4007 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce3709e2416345789234ec12bc6001be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41627 } }
I20260926 13:09:14.907505 4007 leader_election.cc:302] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce3709e2416345789234ec12bc6001be; no voters:
I20260926 13:09:14.907727 4007 leader_election.cc:288] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:14.907819 4012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:14.908031 4007 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:14.908017 4012 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [term 1 LEADER]: Becoming Leader. State: Replica: ce3709e2416345789234ec12bc6001be, State: Running, Role: LEADER
I20260926 13:09:14.908216 4012 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce3709e2416345789234ec12bc6001be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41627 } }
I20260926 13:09:14.908740 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce3709e2416345789234ec12bc6001be. Latest consensus state: current_term: 1 leader_uuid: "ce3709e2416345789234ec12bc6001be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce3709e2416345789234ec12bc6001be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41627 } } }
I20260926 13:09:14.908829 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:14.908978 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce3709e2416345789234ec12bc6001be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce3709e2416345789234ec12bc6001be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41627 } } }
I20260926 13:09:14.909039 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:14.909108 4021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:14.909637 4021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:14.914206 4021 catalog_manager.cc:1347] Generated new cluster ID: e8ebcb05f4e9446881a5d921c87df60f
I20260926 13:09:14.914288 4021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:14.932116 4021 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:14.932708 4021 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:14.947552 4021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce3709e2416345789234ec12bc6001be: Generated new TSK 0
I20260926 13:09:14.947779 4021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260926 13:09:15.106348 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.150s user 0.003s sys 0.000s
I20260926 13:09:15.106673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41627
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:15.186043 4033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:15.186260 4033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:15.186285 4033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:15.187502 4033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:15.187548 4033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:15.187577 4033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:15.189087 4033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:15.189321 4033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:15.189543 4033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:09:15.194765 4033 server_base.cc:1034] running on GCE node
W20260926 13:09:15.194741 4041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:15.194748 4039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:15.194818 4038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:15.195205 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:15.195493 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:15.196645 4033 hybrid_clock.cc:648] HybridClock initialized: now 1790428155196631 us; error 29 us; skew 500 ppm
Sep 26 13:09:15 dist-test-slave-2dw5 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:15.199143 4033 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:09:15.199633 4033 webserver.cc:460] Webserver started at http://127.0.98.1:45727/ using document root <none> and password file <none>
I20260926 13:09:15.199849 4033 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:15.199896 4033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:15.200017 4033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:15.201072 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "e31fe02e77c5433b93a40e2f890e3d95"
format_stamp: "Formatted at 2026-09-26 13:09:15 on dist-test-slave-2dw5"
server_key: "30b27a68442578de92952df2945bf3a6"
server_key_iv: "1a60551f54439516d596ffe80091e35f"
server_key_version: "encryptionkey@0"
I20260926 13:09:15.201413 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e31fe02e77c5433b93a40e2f890e3d95"
format_stamp: "Formatted at 2026-09-26 13:09:15 on dist-test-slave-2dw5"
server_key: "30b27a68442578de92952df2945bf3a6"
server_key_iv: "1a60551f54439516d596ffe80091e35f"
server_key_version: "encryptionkey@0"
I20260926 13:09:15.202663 4033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:15.203488 4048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:15.203655 4033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:09:15.203734 4033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "e31fe02e77c5433b93a40e2f890e3d95"
format_stamp: "Formatted at 2026-09-26 13:09:15 on dist-test-slave-2dw5"
server_key: "30b27a68442578de92952df2945bf3a6"
server_key_iv: "1a60551f54439516d596ffe80091e35f"
server_key_version: "encryptionkey@0"
I20260926 13:09:15.203810 4033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:15.226567 4033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:15.228777 4033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:15.228928 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:15.229140 4033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:15.229457 4033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:15.229490 4033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:15.229519 4033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:15.229537 4033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:15.235327 4033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33591
I20260926 13:09:15.235369 4161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33591 every 8 connection(s)
I20260926 13:09:15.235669 4033 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:09:15 dist-test-slave-2dw5 krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:15.241798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 4033
I20260926 13:09:15.241894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:15.242089 392 external_mini_cluster.cc:1428] Setting key 1a9850426e0f52f4b8bf07d8be71d98c
I20260926 13:09:15.245087 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:15.236387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54015 (local address 127.0.98.62:41627)
0926 13:09:15.236448 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:15.236450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:15.236612 (+ 162us) server_negotiation.cc:407] Connection header received
0926 13:09:15.237560 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:15.237561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:15.237573 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:15.237625 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:15.238239 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.238729 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.239724 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.239929 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.239983 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:15.242129 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:15.242131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:15.242131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:15.242140 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:15.243747 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:15.244168 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:15.244170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:15.244170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:15.244227 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:15.244449 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:15.244450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:15.244451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:15.244567 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:15.244623 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:15.244892 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 13:09:15.244991 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:09:15.245252 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:15.236352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41627 (local address 127.0.98.1:54015)
0926 13:09:15.236494 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:15.236528 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:15.237457 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:15.237698 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:15.237701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:15.237955 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:15.238154 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:15.238158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.238810 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.238811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:15.239636 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:15.239638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.239995 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.239996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:15.240110 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:15.240591 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:15.240599 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:15.242033 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:15.243856 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:15.243863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:15.243864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:15.244088 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:15.244281 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:15.244281 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:15.244282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:15.244394 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:15.244620 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:15.244626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:15.244767 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 13:09:15.244863 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 13:09:15.245028 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260926 13:09:15.245764 4162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41627
I20260926 13:09:15.245854 4162 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:15.246025 4162 heartbeater.cc:511] Master 127.0.98.62:41627 requested a full tablet report, sending...
I20260926 13:09:15.246510 3971 ts_manager.cc:194] Registered new tserver with Master: e31fe02e77c5433b93a40e2f890e3d95 (127.0.98.1:33591)
I20260926 13:09:15.247145 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54015
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:09:15.269857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41627
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:15.353511 4167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:15.353731 4167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:15.353765 4167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:15.354977 4167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:15.355021 4167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:15.355047 4167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:15.356544 4167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:15.356751 4167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:15.356935 4167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:15.361794 4173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:15.361811 4172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:15.361850 4175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:15.361995 4167 server_base.cc:1034] running on GCE node
I20260926 13:09:15.362251 4167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:15.362465 4167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:15.363590 4167 hybrid_clock.cc:648] HybridClock initialized: now 1790428155363572 us; error 35 us; skew 500 ppm
Sep 26 13:09:15 dist-test-slave-2dw5 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:15.366132 4167 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:09:15.366636 4167 webserver.cc:460] Webserver started at http://127.0.98.2:37515/ using document root <none> and password file <none>
I20260926 13:09:15.367056 4167 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:15.367115 4167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:15.367272 4167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:15.368420 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "1cc463ca207f4fc5b922cf485d5d36f8"
format_stamp: "Formatted at 2026-09-26 13:09:15 on dist-test-slave-2dw5"
server_key: "1cbe016379fd71985b1225529f397bf0"
server_key_iv: "5ad9e61085b8d9d4d5250af09ebcb079"
server_key_version: "encryptionkey@0"
I20260926 13:09:15.368750 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1cc463ca207f4fc5b922cf485d5d36f8"
format_stamp: "Formatted at 2026-09-26 13:09:15 on dist-test-slave-2dw5"
server_key: "1cbe016379fd71985b1225529f397bf0"
server_key_iv: "5ad9e61085b8d9d4d5250af09ebcb079"
server_key_version: "encryptionkey@0"
I20260926 13:09:15.369969 4167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:15.370640 4182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:15.370800 4167 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:15.370900 4167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "1cc463ca207f4fc5b922cf485d5d36f8"
format_stamp: "Formatted at 2026-09-26 13:09:15 on dist-test-slave-2dw5"
server_key: "1cbe016379fd71985b1225529f397bf0"
server_key_iv: "5ad9e61085b8d9d4d5250af09ebcb079"
server_key_version: "encryptionkey@0"
I20260926 13:09:15.370960 4167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:15.381989 4167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:15.384078 4167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:15.384225 4167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:15.384443 4167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:15.384733 4167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:15.384768 4167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:15.384797 4167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:15.384820 4167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:15.390259 4167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41233
I20260926 13:09:15.390314 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41233 every 8 connection(s)
I20260926 13:09:15.390615 4167 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:09:15.394604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 4167
I20260926 13:09:15.394726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:15.394951 392 external_mini_cluster.cc:1428] Setting key 36942b4953d75bb271380f78b51351da
Sep 26 13:09:15 dist-test-slave-2dw5 krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:15.401643 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:15.391360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38433 (local address 127.0.98.62:41627)
0926 13:09:15.391441 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:15.391444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:15.391694 (+ 250us) server_negotiation.cc:407] Connection header received
0926 13:09:15.392816 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:15.392817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:15.392829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:15.392887 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:15.393558 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.394089 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.395406 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.395657 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.395712 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:15.398440 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:15.398442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:15.398442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:15.398449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:15.400015 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:15.400562 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:15.400563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:15.400564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:15.400625 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:15.400945 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:15.400946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:15.400947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:15.401074 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:15.401129 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:15.401498 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 13:09:15.401573 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:09:15.401888 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:15.391336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41627 (local address 127.0.98.2:38433)
0926 13:09:15.391530 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:15.391579 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:15.392608 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:15.392977 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:15.392982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:15.393264 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:15.393462 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:15.393467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.394191 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.394192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:15.395326 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:15.395329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.395735 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.395736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:15.395885 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:15.396398 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:15.396408 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:15.398296 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:15.400104 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:15.400115 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:15.400118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:15.400454 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:15.400701 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:15.400702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:15.400703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:15.400862 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:15.401140 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:15.401146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:15.401332 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 13:09:15.401456 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 13:09:15.401634 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260926 13:09:15.402431 4296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41627
I20260926 13:09:15.402540 4296 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:15.402745 4296 heartbeater.cc:511] Master 127.0.98.62:41627 requested a full tablet report, sending...
I20260926 13:09:15.403194 3971 ts_manager.cc:194] Registered new tserver with Master: 1cc463ca207f4fc5b922cf485d5d36f8 (127.0.98.2:41233)
I20260926 13:09:15.403579 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38433
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:09:15.421746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41627
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:15.501958 4301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:15.502180 4301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:15.502204 4301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:15.503387 4301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:15.503444 4301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:15.503471 4301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:15.504904 4301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:15.505095 4301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:15.505278 4301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:15.510389 4307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:15.510468 4309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:15.510509 4301 server_base.cc:1034] running on GCE node
W20260926 13:09:15.510401 4306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:15.510792 4301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:15.511097 4301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:15.512234 4301 hybrid_clock.cc:648] HybridClock initialized: now 1790428155512222 us; error 28 us; skew 500 ppm
Sep 26 13:09:15 dist-test-slave-2dw5 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:15.515008 4301 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:09:15.515525 4301 webserver.cc:460] Webserver started at http://127.0.98.3:34191/ using document root <none> and password file <none>
I20260926 13:09:15.515740 4301 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:15.515790 4301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:15.515980 4301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:15.517110 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c43c0ad07cf471c89bffd2d242ed8a3"
format_stamp: "Formatted at 2026-09-26 13:09:15 on dist-test-slave-2dw5"
server_key: "3a39a05aa4296c4a39aad9265f9ef035"
server_key_iv: "763746d1279b6b6a2203c808868d1a50"
server_key_version: "encryptionkey@0"
I20260926 13:09:15.517455 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c43c0ad07cf471c89bffd2d242ed8a3"
format_stamp: "Formatted at 2026-09-26 13:09:15 on dist-test-slave-2dw5"
server_key: "3a39a05aa4296c4a39aad9265f9ef035"
server_key_iv: "763746d1279b6b6a2203c808868d1a50"
server_key_version: "encryptionkey@0"
I20260926 13:09:15.518759 4301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:09:15.519604 4316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:15.519759 4301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:15.519851 4301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "6c43c0ad07cf471c89bffd2d242ed8a3"
format_stamp: "Formatted at 2026-09-26 13:09:15 on dist-test-slave-2dw5"
server_key: "3a39a05aa4296c4a39aad9265f9ef035"
server_key_iv: "763746d1279b6b6a2203c808868d1a50"
server_key_version: "encryptionkey@0"
I20260926 13:09:15.519917 4301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:15.538057 4301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:15.540262 4301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:15.540405 4301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:15.540628 4301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:15.540942 4301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:15.540977 4301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:15.541005 4301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:15.541023 4301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:15.546730 4301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33031
I20260926 13:09:15.546777 4429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33031 every 8 connection(s)
I20260926 13:09:15.547114 4301 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:09:15 dist-test-slave-2dw5 krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:15.556846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 4301
I20260926 13:09:15.556953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:15.556957 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:15.547816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33145 (local address 127.0.98.62:41627)
0926 13:09:15.547957 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:15.547959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:15.548188 (+ 229us) server_negotiation.cc:407] Connection header received
0926 13:09:15.549280 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:15.549281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:15.549295 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:15.549362 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:15.550162 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.550684 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.551797 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.552004 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.552055 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:15.554440 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:15.554442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:15.554442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:15.554450 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:15.555650 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:15.556044 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:15.556045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:15.556045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:15.556103 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:15.556323 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:15.556324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:15.556324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:15.556444 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:15.556499 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:15.556789 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 13:09:15.556866 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:09:15.557129 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:15.547783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41627 (local address 127.0.98.3:33145)
0926 13:09:15.548063 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:15.548104 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:15.549143 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:15.549445 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:15.549450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:15.549836 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:15.550097 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:15.550102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.550760 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.550761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:15.551720 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:15.551722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.552078 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.552079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:15.552191 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:15.552679 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:15.552687 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:15.554345 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:15.555728 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:15.555735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:15.555738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:15.555969 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:15.556155 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:15.556156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:15.556157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:15.556262 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:15.556497 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:15.556504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:15.556644 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 13:09:15.556748 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 13:09:15.556911 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":90,"threads_started":1}
I20260926 13:09:15.557143 392 external_mini_cluster.cc:1428] Setting key 10138a708e0346601380f30c75b4da1f
I20260926 13:09:15.557579 4430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41627
I20260926 13:09:15.557683 4430 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:15.557874 4430 heartbeater.cc:511] Master 127.0.98.62:41627 requested a full tablet report, sending...
I20260926 13:09:15.558213 3970 ts_manager.cc:194] Registered new tserver with Master: 6c43c0ad07cf471c89bffd2d242ed8a3 (127.0.98.3:33031)
I20260926 13:09:15.558602 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33145
I20260926 13:09:15.569013 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:09:15 dist-test-slave-2dw5 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:09:15 dist-test-slave-2dw5 krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:09:15.592033 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:15.583689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.0.98.1:33591)
0926 13:09:15.583845 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:15.583850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:15.583861 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:15.583886 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:15.583889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:15.583903 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:15.583968 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:15.584603 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.585377 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.586261 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.586593 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.586649 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:15.589049 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:15.589059 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:15.589059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:15.589067 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:15.590540 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:15.591156 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:15.591159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:15.591160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:15.591229 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:15.591486 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:15.591487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:15.591487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:15.591649 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:15.591779 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:15.591879 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 13:09:15.591950 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
Sep 26 13:09:15 dist-test-slave-2dw5 krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:15.600183 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:15.593545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.0.98.62:41627)
0926 13:09:15.593602 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:15.593604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:15.593613 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:15.593648 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:15.593649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:15.593659 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:15.593703 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:15.594210 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.594725 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.595418 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:15.595602 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:15.595656 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:15.597977 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:15.597979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:15.597979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:15.597986 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:15.599151 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:15.599437 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:15.599438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:15.599438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:15.599486 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:15.599687 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:15.599687 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:15.599688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:15.599833 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:15.599915 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:15.600032 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 13:09:15.600094 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 13:09:15.600430 3970 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41746
I20260926 13:09:15.600826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 4033
I20260926 13:09:15.604923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 4167
I20260926 13:09:15.608700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 4301
I20260926 13:09:15.612304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 3939
2026-09-26T13:09:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1955 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:15 dist-test-slave-2dw5 krb5kdc[4442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:15 dist-test-slave-2dw5 krb5kdc[4442](info): set up 2 sockets
Sep 26 13:09:15 dist-test-slave-2dw5 krb5kdc[4442](info): commencing operation
krb5kdc: starting...
W20260926 13:09:16.562927 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:16 dist-test-slave-2dw5 krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:09:16.678064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38243
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:16.757613 4455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:16.757850 4455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:16.757885 4455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:16.759119 4455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:16.759164 4455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:16.759182 4455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:16.759198 4455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:16.759214 4455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:16.760775 4455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:16.761024 4455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:16.761233 4455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:16.766026 4461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:16.766100 4460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:16.766184 4463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:16.766089 4455 server_base.cc:1034] running on GCE node
I20260926 13:09:16.766513 4455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:16.766793 4455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:16.768034 4455 hybrid_clock.cc:648] HybridClock initialized: now 1790428156768013 us; error 36 us; skew 500 ppm
Sep 26 13:09:16 dist-test-slave-2dw5 krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:16.770578 4455 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:09:16.771050 4455 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:16.771109 4455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:16.771234 4455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:16.772317 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "66afce11eff648c7befefc5bc82f230a"
format_stamp: "Formatted at 2026-09-26 13:09:16 on dist-test-slave-2dw5"
server_key: "3ff094e33cec07c94677cdff505dfa98"
server_key_iv: "a1ec08b2ff1e48dbdb4f7a0e8dc5fcb3"
server_key_version: "encryptionkey@0"
I20260926 13:09:16.772642 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "66afce11eff648c7befefc5bc82f230a"
format_stamp: "Formatted at 2026-09-26 13:09:16 on dist-test-slave-2dw5"
server_key: "3ff094e33cec07c94677cdff505dfa98"
server_key_iv: "a1ec08b2ff1e48dbdb4f7a0e8dc5fcb3"
server_key_version: "encryptionkey@0"
I20260926 13:09:16.773968 4455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:16.774713 4469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:16.774906 4455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:16.774989 4455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "66afce11eff648c7befefc5bc82f230a"
format_stamp: "Formatted at 2026-09-26 13:09:16 on dist-test-slave-2dw5"
server_key: "3ff094e33cec07c94677cdff505dfa98"
server_key_iv: "a1ec08b2ff1e48dbdb4f7a0e8dc5fcb3"
server_key_version: "encryptionkey@0"
I20260926 13:09:16.775053 4455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:16.785560 4455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:16.787714 4455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:16.787853 4455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:16.791558 4455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38243
I20260926 13:09:16.791590 4521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38243 every 8 connection(s)
I20260926 13:09:16.791900 4455 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:16.792472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 4455
I20260926 13:09:16.792569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:16.792747 392 external_mini_cluster.cc:1428] Setting key 15dabec916c62de36c5de7d57a77d0b2
I20260926 13:09:16.793035 4522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:16.796311 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a: Bootstrap starting.
I20260926 13:09:16.797011 4522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:16.797263 4522 log.cc:824] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:16.797919 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a: No bootstrap required, opened a new log
Sep 26 13:09:16 dist-test-slave-2dw5 krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:16.799904 4522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66afce11eff648c7befefc5bc82f230a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38243 } }
I20260926 13:09:16.800065 4522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:16.800096 4522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66afce11eff648c7befefc5bc82f230a, State: Initialized, Role: FOLLOWER
I20260926 13:09:16.800223 4522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66afce11eff648c7befefc5bc82f230a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38243 } }
I20260926 13:09:16.800284 4522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:16.800307 4522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:16.800354 4522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:16.801003 4522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66afce11eff648c7befefc5bc82f230a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38243 } }
I20260926 13:09:16.801103 4522 leader_election.cc:302] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66afce11eff648c7befefc5bc82f230a; no voters:
I20260926 13:09:16.801270 4522 leader_election.cc:288] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:16.801331 4527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:16.801530 4522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:16.801548 4527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [term 1 LEADER]: Becoming Leader. State: Replica: 66afce11eff648c7befefc5bc82f230a, State: Running, Role: LEADER
I20260926 13:09:16.801739 4527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66afce11eff648c7befefc5bc82f230a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38243 } }
I20260926 13:09:16.802167 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66afce11eff648c7befefc5bc82f230a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66afce11eff648c7befefc5bc82f230a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38243 } } }
I20260926 13:09:16.802237 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:16.802446 4529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66afce11eff648c7befefc5bc82f230a. Latest consensus state: current_term: 1 leader_uuid: "66afce11eff648c7befefc5bc82f230a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66afce11eff648c7befefc5bc82f230a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38243 } } }
I20260926 13:09:16.802556 4529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:16.802702 4536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:16.803367 4536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:16.803557 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:16.793213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.0.98.62:38243)
0926 13:09:16.793384 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:16.793393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:16.793418 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:09:16.793844 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:16.793850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:16.793902 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:16.794014 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:16.794822 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:16.795841 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:16.796872 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:16.797333 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:16.797440 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:16.799796 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:16.799800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:16.799801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:16.799813 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:16.801734 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:16.802140 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:16.802143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:16.802144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:16.802243 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:16.802456 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:16.802457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:16.802457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:16.802766 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:16.802934 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:16.803213 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 13:09:16.803352 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260926 13:09:16.805163 4536 catalog_manager.cc:1347] Generated new cluster ID: c10908a7066d4ba689138a859fce51ee
I20260926 13:09:16.805219 4536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:16.809331 4536 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:16.809871 4536 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:16.813094 4536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a: Generated new TSK 0
I20260926 13:09:16.813254 4536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:09:16.842824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38243
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:16.924993 4548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:16.925212 4548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:16.925246 4548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:16.926401 4548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:16.926443 4548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:16.926481 4548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:16.928165 4548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:16.928422 4548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:16.928637 4548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:16.933768 4553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:16.933709 4554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:16.933705 4556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:16.934197 4548 server_base.cc:1034] running on GCE node
I20260926 13:09:16.934408 4548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:16.934679 4548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:16.935814 4548 hybrid_clock.cc:648] HybridClock initialized: now 1790428156935801 us; error 33 us; skew 500 ppm
Sep 26 13:09:16 dist-test-slave-2dw5 krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:16.938364 4548 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:09:16.938722 4548 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:16.938771 4548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:16.938920 4548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:16.940006 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "830aa56b21b045a4b4ba8277db893163"
format_stamp: "Formatted at 2026-09-26 13:09:16 on dist-test-slave-2dw5"
server_key: "8bc9b0434edae5d09c782ead5904b956"
server_key_iv: "d881532e5c07fb547fb7a1a3a909b2c5"
server_key_version: "encryptionkey@0"
I20260926 13:09:16.940303 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "830aa56b21b045a4b4ba8277db893163"
format_stamp: "Formatted at 2026-09-26 13:09:16 on dist-test-slave-2dw5"
server_key: "8bc9b0434edae5d09c782ead5904b956"
server_key_iv: "d881532e5c07fb547fb7a1a3a909b2c5"
server_key_version: "encryptionkey@0"
I20260926 13:09:16.941491 4548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:16.942180 4562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:16.942322 4548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:16.942400 4548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "830aa56b21b045a4b4ba8277db893163"
format_stamp: "Formatted at 2026-09-26 13:09:16 on dist-test-slave-2dw5"
server_key: "8bc9b0434edae5d09c782ead5904b956"
server_key_iv: "d881532e5c07fb547fb7a1a3a909b2c5"
server_key_version: "encryptionkey@0"
I20260926 13:09:16.942468 4548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:16.958748 4548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:16.960696 4548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:16.960829 4548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:16.961047 4548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:16.961349 4548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:16.961382 4548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:16.961411 4548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:16.961428 4548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:16.966992 4548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37951
I20260926 13:09:16.967033 4675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37951 every 8 connection(s)
I20260926 13:09:16.967284 4548 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:09:16.968587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 4548
I20260926 13:09:16.968679 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:16.968858 392 external_mini_cluster.cc:1428] Setting key a1e39a6964f0cffab6520487732e937c
Sep 26 13:09:16 dist-test-slave-2dw5 krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:09:16.977541 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:16.967976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50047 (local address 127.0.98.62:38243)
0926 13:09:16.968026 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:16.968028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:16.968265 (+ 237us) server_negotiation.cc:407] Connection header received
0926 13:09:16.968727 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:16.968728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:16.968742 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:16.968821 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:16.969900 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:16.970671 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:16.972126 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:16.972329 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:16.972382 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:16.974646 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:16.974647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:16.974647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:16.974655 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:16.976086 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:16.976489 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:16.976490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:16.976490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:16.976548 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:16.976824 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:16.976825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:16.976825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:16.977013 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:16.977068 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:16.977355 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 13:09:16.977438 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:09:16.977619 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:16.967894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38243 (local address 127.0.98.1:50047)
0926 13:09:16.968124 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:16.968165 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:16.968620 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:16.968878 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:16.968881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:16.969233 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:16.969808 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:16.969813 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:16.970765 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:16.970766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:16.972004 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:16.972007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:16.972403 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:16.972404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:16.972540 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:16.973024 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:16.973034 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:16.974563 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:16.976169 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:16.976175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:16.976177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:16.976422 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:16.976616 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:16.976617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:16.976617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:16.976754 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:16.977104 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:16.977109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:16.977225 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 13:09:16.977309 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 13:09:16.977430 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20260926 13:09:16.978075 4676 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38243
I20260926 13:09:16.978159 4676 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:16.978277 4676 heartbeater.cc:511] Master 127.0.98.62:38243 requested a full tablet report, sending...
I20260926 13:09:16.978960 4486 ts_manager.cc:194] Registered new tserver with Master: 830aa56b21b045a4b4ba8277db893163 (127.0.98.1:37951)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 13:09:16.979751 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:09:16.995211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38243
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:17.075868 4681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:17.076112 4681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:17.076146 4681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:17.077383 4681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:17.077426 4681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:17.077463 4681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:17.079110 4681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:17.079365 4681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:17.079572 4681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:17.084461 4689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:17.084553 4687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:17.084458 4686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:17.084743 4681 server_base.cc:1034] running on GCE node
I20260926 13:09:17.085071 4681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:17.085335 4681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:17.086515 4681 hybrid_clock.cc:648] HybridClock initialized: now 1790428157086501 us; error 31 us; skew 500 ppm
Sep 26 13:09:17 dist-test-slave-2dw5 krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:17.089458 4681 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:09:17.089790 4681 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:17.089840 4681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:17.089988 4681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:17.091081 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c315d27e91f43458f7d749134bf664e"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "826bc719e30c5bfea4562a24368db030"
server_key_iv: "66bb1320e1116c518861a7a1a2531407"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.091396 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c315d27e91f43458f7d749134bf664e"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "826bc719e30c5bfea4562a24368db030"
server_key_iv: "66bb1320e1116c518861a7a1a2531407"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.092569 4681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:17.093210 4695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.093351 4681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:17.093454 4681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "1c315d27e91f43458f7d749134bf664e"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "826bc719e30c5bfea4562a24368db030"
server_key_iv: "66bb1320e1116c518861a7a1a2531407"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.093515 4681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:17.128893 4681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:17.131230 4681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:17.131366 4681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:17.131597 4681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:17.131896 4681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:17.131927 4681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.131956 4681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:17.131973 4681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.137426 4681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38377
I20260926 13:09:17.137499 4808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38377 every 8 connection(s)
I20260926 13:09:17.137754 4681 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:09:17.140090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 4681
I20260926 13:09:17.140178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:17.140353 392 external_mini_cluster.cc:1428] Setting key a841ed33c92671d48e7c000e1ca79a1a
Sep 26 13:09:17 dist-test-slave-2dw5 krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:17.147538 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.138526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47587 (local address 127.0.98.62:38243)
0926 13:09:17.138624 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.138626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.138773 (+ 147us) server_negotiation.cc:407] Connection header received
0926 13:09:17.139382 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.139384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.139397 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.139448 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:17.140526 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.141157 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.142226 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.142432 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.142488 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.144744 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:17.144745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:17.144746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:17.144754 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:17.146096 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:17.146543 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:17.146544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:17.146545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:17.146632 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:17.146895 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:17.146896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:17.146896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:17.147015 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:17.147070 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.147364 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.147446 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 13:09:17.147703 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.138467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38243 (local address 127.0.98.2:47587)
0926 13:09:17.138645 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.138691 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.139274 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.139524 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.139529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.139805 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:17.140360 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.140366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.141269 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.141270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.142132 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.142135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.142502 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.142503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.142667 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.143220 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:17.143228 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:17.144646 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:17.146191 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:17.146199 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:17.146201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:17.146472 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:17.146700 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:17.146701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:17.146701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:17.146810 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:17.147070 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:17.147077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:17.147232 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.147318 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.147478 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 13:09:17.148324 4809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38243
I20260926 13:09:17.148411 4809 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:17.148553 4809 heartbeater.cc:511] Master 127.0.98.62:38243 requested a full tablet report, sending...
I20260926 13:09:17.148880 4486 ts_manager.cc:194] Registered new tserver with Master: 1c315d27e91f43458f7d749134bf664e (127.0.98.2:38377)
I20260926 13:09:17.149257 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:09:17.166559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38243
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:17.247072 4814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:17.247294 4814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:17.247318 4814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:17.248494 4814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:17.248533 4814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:17.248559 4814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:17.250064 4814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:17.250257 4814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:17.250442 4814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:17.255342 4822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:17.255357 4814 server_base.cc:1034] running on GCE node
W20260926 13:09:17.255359 4819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:17.255342 4820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:17.255756 4814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:17.255971 4814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:17.257153 4814 hybrid_clock.cc:648] HybridClock initialized: now 1790428157257139 us; error 30 us; skew 500 ppm
Sep 26 13:09:17 dist-test-slave-2dw5 krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:17.259768 4814 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:09:17.260145 4814 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:17.260200 4814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:17.260321 4814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:17.261413 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "c2dd325acbab442bbb729f1e9fb9d5ee"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "b812a373fbb9c9d03b824ed758eb30ff"
server_key_iv: "b1ee9d8ddfc4ce1bac0c5ed37bf8ce4e"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.261742 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c2dd325acbab442bbb729f1e9fb9d5ee"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "b812a373fbb9c9d03b824ed758eb30ff"
server_key_iv: "b1ee9d8ddfc4ce1bac0c5ed37bf8ce4e"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.263139 4814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:17.263931 4828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.264128 4814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:17.264215 4814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "c2dd325acbab442bbb729f1e9fb9d5ee"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "b812a373fbb9c9d03b824ed758eb30ff"
server_key_iv: "b1ee9d8ddfc4ce1bac0c5ed37bf8ce4e"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.264289 4814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:17.279574 4814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:17.281811 4814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:17.281956 4814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:17.282171 4814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:17.282473 4814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:17.282506 4814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.282534 4814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:17.282558 4814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.288216 4814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42133
I20260926 13:09:17.288255 4941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42133 every 8 connection(s)
I20260926 13:09:17.288513 4814 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:17.291265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 4814
I20260926 13:09:17.291352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:17.291507 392 external_mini_cluster.cc:1428] Setting key 92388959d193e3fa11a864fd72c11ad5
Sep 26 13:09:17 dist-test-slave-2dw5 krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:17.297977 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.289230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59993 (local address 127.0.98.62:38243)
0926 13:09:17.289286 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.289288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.289460 (+ 172us) server_negotiation.cc:407] Connection header received
0926 13:09:17.289991 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.289993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.290003 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.290059 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:17.291075 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.291676 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.292809 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.293019 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.293078 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.295287 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:17.295288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:17.295288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:17.295295 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:17.296679 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:17.297074 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:17.297075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:17.297076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:17.297130 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:17.297361 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:17.297361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:17.297362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:17.297479 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:17.297530 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.297818 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.297905 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:09:17.298098 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.289153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38243 (local address 127.0.98.3:59993)
0926 13:09:17.289346 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.289383 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.289884 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.290128 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.290132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.290412 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:17.290998 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.291004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.291766 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.291767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.292734 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.292737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.293089 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.293090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.293265 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.293769 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:17.293778 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:17.295197 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:17.296752 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:17.296759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:17.296761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:17.296984 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:17.297195 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:17.297196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:17.297196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:17.297299 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:17.297532 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:17.297536 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:17.297669 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.297756 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.297904 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":77,"threads_started":1}
I20260926 13:09:17.298614 4942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38243
I20260926 13:09:17.298717 4942 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:17.298899 4942 heartbeater.cc:511] Master 127.0.98.62:38243 requested a full tablet report, sending...
I20260926 13:09:17.299376 4486 ts_manager.cc:194] Registered new tserver with Master: c2dd325acbab442bbb729f1e9fb9d5ee (127.0.98.3:42133)
I20260926 13:09:17.299778 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59993
I20260926 13:09:17.303283 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:09:17.310303 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.304495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.0.98.62:38243)
0926 13:09:17.304625 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.304626 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.304634 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:17.304717 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.304719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.304730 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.304767 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:17.305408 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.305896 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.306639 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.306833 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.306911 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.308017 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:17.308018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:17.308019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:17.308026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:17.309250 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:17.309583 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:17.309584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:17.309584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:17.309635 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:17.309792 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:17.309793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:17.309793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:17.309938 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:17.310038 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.310135 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.310211 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 13:09:17.311978 4486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:17.313547 4486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:17.320007 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.315971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.0.98.3:42133)
0926 13:09:17.316417 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.316422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.316431 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:17.316467 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.316469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.316484 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.316624 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:17.316707 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.317836 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.319081 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.319605 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.319670 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.319735 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.319875 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.319916 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":85,"threads_started":1}
I20260926 13:09:17.320005 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.315770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42133 (local address 127.0.0.1:56296)
0926 13:09:17.316087 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.316129 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.316214 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.316578 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.316579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.316580 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:17.316658 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.316659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.317944 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.317945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.319013 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.319015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.319690 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.319691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.319773 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.319788 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.319854 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.319912 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":45,"thread_start_us":74,"threads_started":1}
I20260926 13:09:17.320194 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.315997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.0.98.2:38377)
0926 13:09:17.316183 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.316187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.316337 (+ 150us) server_negotiation.cc:407] Connection header received
0926 13:09:17.316380 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.316382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.316398 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.316477 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:17.316893 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.317923 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.318975 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.319672 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.319740 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.319806 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.320055 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.320097 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260926 13:09:17.320358 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.316259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.0.98.1:37951)
0926 13:09:17.316416 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.316421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.316430 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:17.316464 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.316466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.316479 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.316730 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:17.316812 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.317501 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.318922 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.319672 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.319740 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.319795 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.320055 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.320228 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260926 13:09:17.320453 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.315770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38377 (local address 127.0.0.1:34062)
0926 13:09:17.316221 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.316230 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.316274 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.316518 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.316521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.316523 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:17.316814 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.316815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.318004 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.318005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.318888 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.318896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.319833 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.319834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.319915 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.319921 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.319968 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.320013 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":6,"thread_start_us":62,"threads_started":1}
I20260926 13:09:17.320574 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.315969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37951 (local address 127.0.0.1:59476)
0926 13:09:17.316252 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.316262 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.316312 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.316570 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.316570 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.316572 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:17.316649 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.316651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.317591 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.317592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.318809 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.318812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.319830 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.319831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.319915 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.319923 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.319968 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.320012 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":28,"thread_start_us":137,"threads_started":1}
I20260926 13:09:17.320868 4876 tablet_service.cc:1467] Processing CreateTablet for tablet 0bde5d2f119a4f2fabd92ddf45725d70 (DEFAULT_TABLE table=test-table [id=c0b6eb7f102541eeb2875e7807d6a347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:17.321259 4876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bde5d2f119a4f2fabd92ddf45725d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:17.323506 4743 tablet_service.cc:1467] Processing CreateTablet for tablet 0bde5d2f119a4f2fabd92ddf45725d70 (DEFAULT_TABLE table=test-table [id=c0b6eb7f102541eeb2875e7807d6a347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:17.323606 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 0bde5d2f119a4f2fabd92ddf45725d70 (DEFAULT_TABLE table=test-table [id=c0b6eb7f102541eeb2875e7807d6a347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:17.323827 4743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bde5d2f119a4f2fabd92ddf45725d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:17.323962 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bde5d2f119a4f2fabd92ddf45725d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:17.326311 4961 tablet_bootstrap.cc:492] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163: Bootstrap starting.
I20260926 13:09:17.326237 4960 tablet_bootstrap.cc:492] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e: Bootstrap starting.
I20260926 13:09:17.326588 4959 tablet_bootstrap.cc:492] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee: Bootstrap starting.
I20260926 13:09:17.327016 4961 tablet_bootstrap.cc:654] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:17.327044 4960 tablet_bootstrap.cc:654] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:17.327203 4959 tablet_bootstrap.cc:654] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:17.327526 4960 log.cc:824] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:17.327586 4961 log.cc:824] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:17.327729 4959 log.cc:824] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:17.328167 4960 tablet_bootstrap.cc:492] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e: No bootstrap required, opened a new log
I20260926 13:09:17.328223 4960 ts_tablet_manager.cc:1397] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:17.328281 4961 tablet_bootstrap.cc:492] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163: No bootstrap required, opened a new log
I20260926 13:09:17.328344 4961 ts_tablet_manager.cc:1397] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:09:17.329593 4960 raft_consensus.cc:348] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c315d27e91f43458f7d749134bf664e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38377 } } peers { permanent_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42133 } } peers { permanent_uuid: "830aa56b21b045a4b4ba8277db893163" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37951 } }
I20260926 13:09:17.329737 4960 raft_consensus.cc:374] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:17.329666 4961 raft_consensus.cc:348] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c315d27e91f43458f7d749134bf664e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38377 } } peers { permanent_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42133 } } peers { permanent_uuid: "830aa56b21b045a4b4ba8277db893163" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37951 } }
I20260926 13:09:17.329771 4960 raft_consensus.cc:729] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c315d27e91f43458f7d749134bf664e, State: Initialized, Role: FOLLOWER
I20260926 13:09:17.329792 4961 raft_consensus.cc:374] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:17.329825 4961 raft_consensus.cc:729] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830aa56b21b045a4b4ba8277db893163, State: Initialized, Role: FOLLOWER
I20260926 13:09:17.329885 4960 consensus_queue.cc:260] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c315d27e91f43458f7d749134bf664e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38377 } } peers { permanent_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42133 } } peers { permanent_uuid: "830aa56b21b045a4b4ba8277db893163" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37951 } }
I20260926 13:09:17.329924 4961 consensus_queue.cc:260] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c315d27e91f43458f7d749134bf664e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38377 } } peers { permanent_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42133 } } peers { permanent_uuid: "830aa56b21b045a4b4ba8277db893163" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37951 } }
I20260926 13:09:17.330158 4960 ts_tablet_manager.cc:1428] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:09:17.330179 4961 ts_tablet_manager.cc:1428] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:09:17.330269 4809 heartbeater.cc:503] Master 127.0.98.62:38243 was elected leader, sending a full tablet report...
I20260926 13:09:17.330278 4676 heartbeater.cc:503] Master 127.0.98.62:38243 was elected leader, sending a full tablet report...
I20260926 13:09:17.331133 4959 tablet_bootstrap.cc:492] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee: No bootstrap required, opened a new log
I20260926 13:09:17.331271 4959 ts_tablet_manager.cc:1397] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260926 13:09:17.332662 4959 raft_consensus.cc:348] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c315d27e91f43458f7d749134bf664e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38377 } } peers { permanent_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42133 } } peers { permanent_uuid: "830aa56b21b045a4b4ba8277db893163" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37951 } }
I20260926 13:09:17.332803 4959 raft_consensus.cc:374] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:17.332841 4959 raft_consensus.cc:729] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dd325acbab442bbb729f1e9fb9d5ee, State: Initialized, Role: FOLLOWER
I20260926 13:09:17.332955 4959 consensus_queue.cc:260] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c315d27e91f43458f7d749134bf664e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38377 } } peers { permanent_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42133 } } peers { permanent_uuid: "830aa56b21b045a4b4ba8277db893163" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37951 } }
I20260926 13:09:17.333192 4959 ts_tablet_manager.cc:1428] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 13:09:17.333220 4942 heartbeater.cc:503] Master 127.0.98.62:38243 was elected leader, sending a full tablet report...
W20260926 13:09:17.388373 4810 tablet.cc:2369] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:09:17.424677 4967 raft_consensus.cc:482] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:17.424813 4967 raft_consensus.cc:504] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c315d27e91f43458f7d749134bf664e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38377 } } peers { permanent_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42133 } } peers { permanent_uuid: "830aa56b21b045a4b4ba8277db893163" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37951 } }
I20260926 13:09:17.425172 4967 leader_election.cc:288] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c315d27e91f43458f7d749134bf664e (127.0.98.2:38377), 830aa56b21b045a4b4ba8277db893163 (127.0.98.1:37951)
I20260926 13:09:17.428233 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.425478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47959 (local address 127.0.98.2:38377)
0926 13:09:17.425528 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.425530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.425541 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:17.425571 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.425572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.425584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.425661 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:17.425819 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.426360 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.427462 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.427896 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.427947 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.427955 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.428136 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.428166 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:09:17.428239 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.425372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38377 (local address 127.0.98.3:47959)
0926 13:09:17.425421 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.425432 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.425476 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.425677 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.425678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.425680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:17.425749 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.425752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.426476 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.426477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.427353 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.427355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.427974 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.427974 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.428059 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.428063 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.428123 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.428166 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 13:09:17.428488 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.425468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37951 (local address 127.0.98.3:42469)
0926 13:09:17.425632 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.425643 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.425697 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.425818 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.425819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.425820 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:17.425881 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.425882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.426716 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.426717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.427617 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:17.427620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.428258 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.428259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:17.428323 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.428327 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.428379 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.428419 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
I20260926 13:09:17.428506 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.425545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42469 (local address 127.0.98.1:37951)
0926 13:09:17.425595 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.425597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.425704 (+ 107us) server_negotiation.cc:407] Connection header received
0926 13:09:17.425740 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.425741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.425754 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.425809 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:17.426100 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.426622 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.427709 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.428109 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.428222 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.428231 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.428387 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.428452 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:09:17.428574 4763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bde5d2f119a4f2fabd92ddf45725d70" candidate_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c315d27e91f43458f7d749134bf664e" is_pre_election: true
I20260926 13:09:17.428800 4763 raft_consensus.cc:2454] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2dd325acbab442bbb729f1e9fb9d5ee in term 0.
I20260926 13:09:17.429023 4832 leader_election.cc:302] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c315d27e91f43458f7d749134bf664e, c2dd325acbab442bbb729f1e9fb9d5ee; no voters:
I20260926 13:09:17.429174 4967 raft_consensus.cc:2781] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:17.429172 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bde5d2f119a4f2fabd92ddf45725d70" candidate_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830aa56b21b045a4b4ba8277db893163" is_pre_election: true
I20260926 13:09:17.429242 4967 raft_consensus.cc:482] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:17.429260 4967 raft_consensus.cc:3037] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:17.429293 4630 raft_consensus.cc:2454] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2dd325acbab442bbb729f1e9fb9d5ee in term 0.
I20260926 13:09:17.429948 4967 raft_consensus.cc:504] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c315d27e91f43458f7d749134bf664e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38377 } } peers { permanent_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42133 } } peers { permanent_uuid: "830aa56b21b045a4b4ba8277db893163" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37951 } }
I20260926 13:09:17.430075 4967 leader_election.cc:288] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [CANDIDATE]: Term 1 election: Requested vote from peers 1c315d27e91f43458f7d749134bf664e (127.0.98.2:38377), 830aa56b21b045a4b4ba8277db893163 (127.0.98.1:37951)
I20260926 13:09:17.430277 4763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bde5d2f119a4f2fabd92ddf45725d70" candidate_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c315d27e91f43458f7d749134bf664e"
I20260926 13:09:17.430315 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bde5d2f119a4f2fabd92ddf45725d70" candidate_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830aa56b21b045a4b4ba8277db893163"
I20260926 13:09:17.430344 4763 raft_consensus.cc:3037] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:17.430380 4630 raft_consensus.cc:3037] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:17.431118 4763 raft_consensus.cc:2454] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2dd325acbab442bbb729f1e9fb9d5ee in term 1.
I20260926 13:09:17.431159 4630 raft_consensus.cc:2454] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2dd325acbab442bbb729f1e9fb9d5ee in term 1.
I20260926 13:09:17.431268 4832 leader_election.cc:302] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c315d27e91f43458f7d749134bf664e, c2dd325acbab442bbb729f1e9fb9d5ee; no voters:
I20260926 13:09:17.431360 4967 raft_consensus.cc:2781] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:17.431553 4967 raft_consensus.cc:686] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 1 LEADER]: Becoming Leader. State: Replica: c2dd325acbab442bbb729f1e9fb9d5ee, State: Running, Role: LEADER
I20260926 13:09:17.431639 4967 consensus_queue.cc:237] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c315d27e91f43458f7d749134bf664e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38377 } } peers { permanent_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42133 } } peers { permanent_uuid: "830aa56b21b045a4b4ba8277db893163" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37951 } }
I20260926 13:09:17.432512 4486 catalog_manager.cc:5495] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee reported cstate change: term changed from 0 to 1, leader changed from <none> to c2dd325acbab442bbb729f1e9fb9d5ee (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c315d27e91f43458f7d749134bf664e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2dd325acbab442bbb729f1e9fb9d5ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "830aa56b21b045a4b4ba8277db893163" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20260926 13:09:17.437538 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.435243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.0.98.3:42133)
0926 13:09:17.435279 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.435281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.435289 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:17.435346 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.435347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.435361 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.435417 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:17.435507 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.435943 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.436794 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:17.436978 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:17.437027 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:17.437225 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:17.437323 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:17.437382 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.437470 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.437487 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:09:17.439303 4763 raft_consensus.cc:1261] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e [term 1 FOLLOWER]: Refusing update from remote peer c2dd325acbab442bbb729f1e9fb9d5ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:17.439303 4630 raft_consensus.cc:1261] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163 [term 1 FOLLOWER]: Refusing update from remote peer c2dd325acbab442bbb729f1e9fb9d5ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:17.439656 4967 consensus_queue.cc:1035] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c315d27e91f43458f7d749134bf664e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:17.439810 4969 consensus_queue.cc:1035] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "830aa56b21b045a4b4ba8277db893163" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:17.442806 4974 mvcc.cc:205] Tried to move back new op lower bound from 7333593732868513792 to 7333593732840591360. Current Snapshot: MvccSnapshot[applied={T|T < 7333593732868513792}]
I20260926 13:09:17.444257 4486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:17.444339 4486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:17.445032 4486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66afce11eff648c7befefc5bc82f230a: Sending DeleteTablet for 3 replicas of tablet 0bde5d2f119a4f2fabd92ddf45725d70
I20260926 13:09:17.445351 4743 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bde5d2f119a4f2fabd92ddf45725d70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34062
I20260926 13:09:17.445389 4876 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bde5d2f119a4f2fabd92ddf45725d70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56296
I20260926 13:09:17.445430 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bde5d2f119a4f2fabd92ddf45725d70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59476
I20260926 13:09:17.445616 4984 tablet_replica.cc:330] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163: stopping tablet replica
I20260926 13:09:17.445706 4984 raft_consensus.cc:2229] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:17.445807 4984 raft_consensus.cc:2258] T 0bde5d2f119a4f2fabd92ddf45725d70 P 830aa56b21b045a4b4ba8277db893163 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:17.445902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 4548
I20260926 13:09:17.445955 4983 tablet_replica.cc:330] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee: stopping tablet replica
I20260926 13:09:17.446028 4983 raft_consensus.cc:2229] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:09:17.446103 4983 raft_consensus.cc:2258] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:17.446439 4982 tablet_replica.cc:330] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e: stopping tablet replica
I20260926 13:09:17.446527 4982 raft_consensus.cc:2229] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:17.446607 4982 raft_consensus.cc:2258] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:17.447175 4982 ts_tablet_manager.cc:1905] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:17.447645 4983 ts_tablet_manager.cc:1905] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:17.450513 4982 ts_tablet_manager.cc:1918] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:17.450592 4982 log.cc:1196] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/0bde5d2f119a4f2fabd92ddf45725d70
I20260926 13:09:17.450829 4982 ts_tablet_manager.cc:1939] T 0bde5d2f119a4f2fabd92ddf45725d70 P 1c315d27e91f43458f7d749134bf664e: Deleting consensus metadata
I20260926 13:09:17.451253 4983 ts_tablet_manager.cc:1918] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:17.451334 4983 log.cc:1196] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/0bde5d2f119a4f2fabd92ddf45725d70
I20260926 13:09:17.451316 4473 catalog_manager.cc:4841] TS 1c315d27e91f43458f7d749134bf664e (127.0.98.2:38377): tablet 0bde5d2f119a4f2fabd92ddf45725d70 (table test-table [id=c0b6eb7f102541eeb2875e7807d6a347]) successfully deleted
I20260926 13:09:17.451565 4983 ts_tablet_manager.cc:1939] T 0bde5d2f119a4f2fabd92ddf45725d70 P c2dd325acbab442bbb729f1e9fb9d5ee: Deleting consensus metadata
I20260926 13:09:17.451942 4472 catalog_manager.cc:4841] TS c2dd325acbab442bbb729f1e9fb9d5ee (127.0.98.3:42133): tablet 0bde5d2f119a4f2fabd92ddf45725d70 (table test-table [id=c0b6eb7f102541eeb2875e7807d6a347]) successfully deleted
W20260926 13:09:17.452912 4473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37951 (error 108)
I20260926 13:09:17.453109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 4681
I20260926 13:09:17.455660 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.455520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37951 (local address 127.0.0.1:59484)
0926 13:09:17.455576 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.455618 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 13:09:17.455794 4473 catalog_manager.cc:4584] TS 830aa56b21b045a4b4ba8277db893163 (127.0.98.1:37951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0bde5d2f119a4f2fabd92ddf45725d70: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37951: connect: Connection refused (error 111)
I20260926 13:09:17.457494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 4814
I20260926 13:09:17.461784 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 4455
2026-09-26T13:09:17Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1849 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T13:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:17Z Disabled control of system clock
I20260926 13:09:17.486781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38907
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 13:09:17.569330 4989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:17.569569 4989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:17.569593 4989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:17.570815 4989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:17.570899 4989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:17.570925 4989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:17.570940 4989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:17.570952 4989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:17.572381 4989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38907
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:17.572654 4989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:17.572970 4989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:17.578039 4995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:17.578220 4994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:17.578222 4997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:17.578303 4989 server_base.cc:1034] running on GCE node
I20260926 13:09:17.578722 4989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:17.579046 4989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:17.580199 4989 hybrid_clock.cc:648] HybridClock initialized: now 1790428157580185 us; error 30 us; skew 500 ppm
I20260926 13:09:17.581607 4989 webserver.cc:460] Webserver started at http://127.0.98.62:37959/ using document root <none> and password file <none>
I20260926 13:09:17.581853 4989 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:17.581902 4989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:17.582037 4989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:17.583240 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "de30b1c06da647b69c75fbb64a40eae9"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "429105614c74568ec1385b0212f59a3d"
server_key_iv: "d8b53bce14f3f9da1fe66e0e19d1b222"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.583571 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "de30b1c06da647b69c75fbb64a40eae9"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "429105614c74568ec1385b0212f59a3d"
server_key_iv: "d8b53bce14f3f9da1fe66e0e19d1b222"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.584908 4989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:17.585825 5003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.586097 4989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:17.586198 4989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "de30b1c06da647b69c75fbb64a40eae9"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "429105614c74568ec1385b0212f59a3d"
server_key_iv: "d8b53bce14f3f9da1fe66e0e19d1b222"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.586279 4989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:17.589723 4989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:17.589962 4989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:17.590076 4989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:17.593777 4989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38907
I20260926 13:09:17.593802 5055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38907 every 8 connection(s)
I20260926 13:09:17.594128 4989 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:17.594801 5056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:17.597738 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9: Bootstrap starting.
I20260926 13:09:17.598335 5056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:17.598609 5056 log.cc:824] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:17.599327 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9: No bootstrap required, opened a new log
I20260926 13:09:17.600925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 4989
I20260926 13:09:17.601002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:17.601161 392 external_mini_cluster.cc:1428] Setting key 68bb2f4b665e7ca4eb12712838dfb017
I20260926 13:09:17.601035 5056 raft_consensus.cc:348] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de30b1c06da647b69c75fbb64a40eae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38907 } }
I20260926 13:09:17.601191 5056 raft_consensus.cc:374] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:17.601229 5056 raft_consensus.cc:729] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de30b1c06da647b69c75fbb64a40eae9, State: Initialized, Role: FOLLOWER
I20260926 13:09:17.601368 5056 consensus_queue.cc:260] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de30b1c06da647b69c75fbb64a40eae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38907 } }
I20260926 13:09:17.601428 5056 raft_consensus.cc:388] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:17.601452 5056 raft_consensus.cc:482] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:17.601497 5056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:17.602110 5056 raft_consensus.cc:504] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de30b1c06da647b69c75fbb64a40eae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38907 } }
I20260926 13:09:17.602205 5056 leader_election.cc:302] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de30b1c06da647b69c75fbb64a40eae9; no voters:
I20260926 13:09:17.602352 5056 leader_election.cc:288] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:17.602415 5061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:17.602528 5061 raft_consensus.cc:686] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [term 1 LEADER]: Becoming Leader. State: Replica: de30b1c06da647b69c75fbb64a40eae9, State: Running, Role: LEADER
I20260926 13:09:17.602595 5056 sys_catalog.cc:564] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:17.602632 5061 consensus_queue.cc:237] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de30b1c06da647b69c75fbb64a40eae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38907 } }
I20260926 13:09:17.603072 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de30b1c06da647b69c75fbb64a40eae9. Latest consensus state: current_term: 1 leader_uuid: "de30b1c06da647b69c75fbb64a40eae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de30b1c06da647b69c75fbb64a40eae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38907 } } }
I20260926 13:09:17.603147 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:17.603353 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.601647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.0.98.62:38907)
0926 13:09:17.601867 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.601874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.601898 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:09:17.601998 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.602003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.602037 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.602206 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:17.602865 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:17.602869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:17.602869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:17.602873 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:17.602918 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:17.602931 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:17.602989 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.603105 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.603171 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260926 13:09:17.603338 5069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:17.604099 5069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:17.603334 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de30b1c06da647b69c75fbb64a40eae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de30b1c06da647b69c75fbb64a40eae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38907 } } }
I20260926 13:09:17.604777 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:17.605610 5069 catalog_manager.cc:1347] Generated new cluster ID: 10415d9b01064e449b6c2a4c902564b7
I20260926 13:09:17.605660 5069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:17.613351 5069 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:17.613526 5069 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:17.619576 5069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9: Generated new TSK 0
I20260926 13:09:17.619823 5069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:09:17.621827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38907
--builtin_ntp_servers=127.0.98.20:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 13:09:17.702260 5080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:17.702466 5080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:17.702491 5080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:17.703750 5080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:17.703792 5080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:17.703819 5080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:17.705188 5080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:17.705418 5080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:17.705698 5080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:17.710371 5086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:17.710435 5085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:17.710556 5088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:17.710880 5080 server_base.cc:1034] running on GCE node
I20260926 13:09:17.711097 5080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:17.711372 5080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:17.712558 5080 hybrid_clock.cc:648] HybridClock initialized: now 1790428157712541 us; error 35 us; skew 500 ppm
I20260926 13:09:17.713972 5080 webserver.cc:460] Webserver started at http://127.0.98.1:36001/ using document root <none> and password file <none>
I20260926 13:09:17.714380 5080 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:17.714498 5080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:17.714649 5080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:17.716205 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "01d021d42d0f490f800e6ef54f00d608"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "236b86a484412fa979985bbefe31b010"
server_key_iv: "d94ca1c711f836c00d3dd99c81da23bb"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.716642 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "01d021d42d0f490f800e6ef54f00d608"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "236b86a484412fa979985bbefe31b010"
server_key_iv: "d94ca1c711f836c00d3dd99c81da23bb"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.718400 5080 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:09:17.719432 5094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.719647 5080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:09:17.719745 5080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "01d021d42d0f490f800e6ef54f00d608"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "236b86a484412fa979985bbefe31b010"
server_key_iv: "d94ca1c711f836c00d3dd99c81da23bb"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.719830 5080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:17.724803 5080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:17.725137 5080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:17.725286 5080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:17.725553 5080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:17.725917 5080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:17.725968 5080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.726006 5080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:17.726037 5080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.733574 5080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44459
I20260926 13:09:17.733631 5207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44459 every 8 connection(s)
I20260926 13:09:17.733997 5080 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:09:17.736536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 5080
I20260926 13:09:17.736652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:17.736838 392 external_mini_cluster.cc:1428] Setting key 0941ac8eae6b058353b27194d41b9a3a
I20260926 13:09:17.737628 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.734704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38377 (local address 127.0.98.62:38907)
0926 13:09:17.734762 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.734764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.735018 (+ 254us) server_negotiation.cc:407] Connection header received
0926 13:09:17.735508 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.735509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.735517 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.735606 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:17.737151 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:17.737153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:17.737153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:17.737158 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:17.737176 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:17.737181 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:17.737235 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.737527 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.737544 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:09:17.737845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38907
--builtin_ntp_servers=127.0.98.20:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 13:09:17.738060 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.734663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38907 (local address 127.0.98.1:38377)
0926 13:09:17.734890 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.734931 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.735401 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.735661 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.735664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.735673 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:17.736977 (+ 1304us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:17.736978 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:17.737019 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:17.737057 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:17.737431 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:17.737437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:17.737439 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.737764 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.737835 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
I20260926 13:09:17.738483 5208 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38907
I20260926 13:09:17.738579 5208 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:17.738726 5208 heartbeater.cc:511] Master 127.0.98.62:38907 requested a full tablet report, sending...
I20260926 13:09:17.739260 5020 ts_manager.cc:194] Registered new tserver with Master: 01d021d42d0f490f800e6ef54f00d608 (127.0.98.1:44459)
W20260926 13:09:17.820096 5211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:17.820312 5211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:17.820333 5211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:17.821503 5211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:17.821540 5211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:17.821565 5211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:17.822965 5211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:17.823175 5211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:17.823431 5211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:17.828795 5219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:17.828812 5211 server_base.cc:1034] running on GCE node
W20260926 13:09:17.828795 5216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:17.828830 5217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:17.829334 5211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:17.829569 5211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:17.830722 5211 hybrid_clock.cc:648] HybridClock initialized: now 1790428157830706 us; error 28 us; skew 500 ppm
I20260926 13:09:17.832018 5211 webserver.cc:460] Webserver started at http://127.0.98.2:42799/ using document root <none> and password file <none>
I20260926 13:09:17.832302 5211 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:17.832353 5211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:17.832474 5211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:17.833540 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "f636bf2010734ab9b86be284af9d43f1"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "21332a90ba4ed74d4ee29556295bb0db"
server_key_iv: "a2b607d59149627e7a911b7ead041b8f"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.833865 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f636bf2010734ab9b86be284af9d43f1"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "21332a90ba4ed74d4ee29556295bb0db"
server_key_iv: "a2b607d59149627e7a911b7ead041b8f"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.835273 5211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:09:17.835999 5225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.836179 5211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.836259 5211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "f636bf2010734ab9b86be284af9d43f1"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "21332a90ba4ed74d4ee29556295bb0db"
server_key_iv: "a2b607d59149627e7a911b7ead041b8f"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.836323 5211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:17.839869 5211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:17.840137 5211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:17.840265 5211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:17.840451 5211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:17.840787 5211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:17.840830 5211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.840864 5211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:17.840888 5211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.846326 5211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34243
I20260926 13:09:17.846756 5211 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:09:17.846735 5338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34243 every 8 connection(s)
I20260926 13:09:17.850337 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.847878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40579 (local address 127.0.98.62:38907)
0926 13:09:17.847926 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.847928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.848067 (+ 139us) server_negotiation.cc:407] Connection header received
0926 13:09:17.848714 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.848716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.848722 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.848773 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:17.849981 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:17.849983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:17.849983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:17.849988 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:17.850006 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:17.850011 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:17.850068 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.850144 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.850160 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:09:17.850436 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.847594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38907 (local address 127.0.98.2:40579)
0926 13:09:17.847919 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.847967 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.848612 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.848846 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.848850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.848860 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:17.849827 (+ 967us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:17.849827 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:17.849857 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:17.849890 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:17.850074 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:17.850080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:17.850083 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.850156 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.850223 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
I20260926 13:09:17.851099 5339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38907
I20260926 13:09:17.851193 5339 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:17.851320 5339 heartbeater.cc:511] Master 127.0.98.62:38907 requested a full tablet report, sending...
I20260926 13:09:17.851641 5020 ts_manager.cc:194] Registered new tserver with Master: f636bf2010734ab9b86be284af9d43f1 (127.0.98.2:34243)
I20260926 13:09:17.852655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 5211
I20260926 13:09:17.852746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:17.852907 392 external_mini_cluster.cc:1428] Setting key 0b1900ba9064fd6764c8bf7c03719af1
I20260926 13:09:17.853812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38907
--builtin_ntp_servers=127.0.98.20:34397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 13:09:17.934945 5342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:17.935184 5342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:17.935209 5342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:17.936486 5342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:17.936539 5342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:17.936568 5342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:17.938046 5342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:17.938323 5342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:17.938620 5342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:17.943756 5347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:17.943774 5350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:17.943758 5348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:17.943851 5342 server_base.cc:1034] running on GCE node
I20260926 13:09:17.944201 5342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:17.944478 5342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:17.945622 5342 hybrid_clock.cc:648] HybridClock initialized: now 1790428157945607 us; error 32 us; skew 500 ppm
I20260926 13:09:17.946914 5342 webserver.cc:460] Webserver started at http://127.0.98.3:33973/ using document root <none> and password file <none>
I20260926 13:09:17.947137 5342 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:17.947186 5342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:17.947314 5342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:17.948539 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "b17bddc45b094c8a985d14574cf0d30f"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "f7c83e988bb3b6d1fa4e3d7e72818b12"
server_key_iv: "3aa43da421f9eb83c6c3f2ef2ce4fe27"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.948885 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b17bddc45b094c8a985d14574cf0d30f"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "f7c83e988bb3b6d1fa4e3d7e72818b12"
server_key_iv: "3aa43da421f9eb83c6c3f2ef2ce4fe27"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.950153 5342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:09:17.950982 5356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.951170 5342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:17.951264 5342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "b17bddc45b094c8a985d14574cf0d30f"
format_stamp: "Formatted at 2026-09-26 13:09:17 on dist-test-slave-2dw5"
server_key: "f7c83e988bb3b6d1fa4e3d7e72818b12"
server_key_iv: "3aa43da421f9eb83c6c3f2ef2ce4fe27"
server_key_version: "encryptionkey@0"
I20260926 13:09:17.951344 5342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:17.954914 5342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:17.955219 5342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:17.955350 5342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:17.955547 5342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:17.955879 5342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:17.955917 5342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.955957 5342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:17.955984 5342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:17.961908 5342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46721
I20260926 13:09:17.961969 5469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46721 every 8 connection(s)
I20260926 13:09:17.962291 5342 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:17.965363 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.962989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56183 (local address 127.0.98.62:38907)
0926 13:09:17.963052 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.963054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.963316 (+ 262us) server_negotiation.cc:407] Connection header received
0926 13:09:17.963853 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.963854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.963862 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.963927 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:17.965097 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:17.965099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:17.965099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:17.965105 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:17.965122 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:17.965127 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:17.965187 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.965269 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.965285 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:09:17.965570 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.962983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38907 (local address 127.0.98.3:56183)
0926 13:09:17.963191 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.963225 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.963743 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.964002 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.964006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.964016 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:17.964929 (+ 913us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:17.964930 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:17.964963 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:17.964994 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:17.965188 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:17.965193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:17.965196 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.965275 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.965339 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260926 13:09:17.966148 5470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38907
I20260926 13:09:17.966233 5470 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:17.966346 5470 heartbeater.cc:511] Master 127.0.98.62:38907 requested a full tablet report, sending...
I20260926 13:09:17.966681 5020 ts_manager.cc:194] Registered new tserver with Master: b17bddc45b094c8a985d14574cf0d30f (127.0.98.3:46721)
I20260926 13:09:17.968508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 5342
I20260926 13:09:17.968598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:17.968768 392 external_mini_cluster.cc:1428] Setting key dde214b2a1999cfbd064175458aba138
I20260926 13:09:17.969452 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:09:17.971835 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.970753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.0.98.62:38907)
0926 13:09:17.970833 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.970835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.970867 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:09:17.970934 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.970934 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.970939 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.970995 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:17.971597 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:17.971599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:17.971599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:17.971604 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:17.971622 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:17.971628 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:17.971682 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.971745 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.971762 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 13:09:17.973491 5020 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:17.975102 5020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:17.978782 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.977295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34243 (local address 127.0.0.1:54300)
0926 13:09:17.977608 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.977646 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.977700 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.977881 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.977886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.977893 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:17.978380 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:17.978381 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:17.978397 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:17.978421 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:17.978644 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:17.978651 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:17.978654 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.978700 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.978712 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":110,"threads_started":1}
I20260926 13:09:17.978957 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.977383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.0.98.3:46721)
0926 13:09:17.977562 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.977568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.977763 (+ 195us) server_negotiation.cc:407] Connection header received
0926 13:09:17.977787 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.977790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.977800 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.977861 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:17.978586 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:17.978593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:17.978594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:17.978598 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:17.978631 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:17.978657 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:17.978713 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.978829 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.978875 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":122,"threads_started":1}
I20260926 13:09:17.978928 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.977291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46721 (local address 127.0.0.1:47912)
0926 13:09:17.977603 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.977645 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.977706 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.978172 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.978172 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.978175 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:17.978502 (+ 327us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:17.978502 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:17.978509 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:17.978520 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:17.978738 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:17.978739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:17.978740 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.978781 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.978794 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":57,"threads_started":1}
I20260926 13:09:17.979377 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.977483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44459 (local address 127.0.0.1:35206)
0926 13:09:17.977849 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:17.977857 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:17.977897 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:17.978505 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:17.978506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:17.978509 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:17.978934 (+ 425us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:17.978935 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:17.978940 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:17.978950 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:17.979268 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:17.979269 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:17.979269 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 13:09:17.979310 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 13:09:17.979321 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":69,"threads_started":1}
I20260926 13:09:17.979400 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.977400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54300 (local address 127.0.98.2:34243)
0926 13:09:17.977687 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.977691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.977702 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:17.977755 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.977759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.977769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.977870 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:17.978499 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:17.978505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:17.978505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:17.978509 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:17.978540 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:17.978560 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:17.979268 (+ 708us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.979291 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.979308 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":72,"threads_started":1}
I20260926 13:09:17.979564 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:17.977693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35206 (local address 127.0.98.1:44459)
0926 13:09:17.977831 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:17.977835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:17.977942 (+ 107us) server_negotiation.cc:407] Connection header received
0926 13:09:17.977979 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:17.977981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:17.977990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:17.978053 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:17.979026 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:17.979034 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:17.979034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:17.979038 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:17.979072 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:17.979096 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:17.979240 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:17.979469 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 13:09:17.979485 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260926 13:09:17.979848 5404 tablet_service.cc:1467] Processing CreateTablet for tablet 27977c8902c44a21979b0b74ede0cef6 (DEFAULT_TABLE table=test-table [id=0a637f52413b45d9b979558135293d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:17.980211 5404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27977c8902c44a21979b0b74ede0cef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:17.980197 5273 tablet_service.cc:1467] Processing CreateTablet for tablet 27977c8902c44a21979b0b74ede0cef6 (DEFAULT_TABLE table=test-table [id=0a637f52413b45d9b979558135293d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:17.980255 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 27977c8902c44a21979b0b74ede0cef6 (DEFAULT_TABLE table=test-table [id=0a637f52413b45d9b979558135293d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:17.980482 5273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27977c8902c44a21979b0b74ede0cef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:17.980507 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27977c8902c44a21979b0b74ede0cef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:17.983362 5484 tablet_bootstrap.cc:492] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608: Bootstrap starting.
I20260926 13:09:17.983685 5486 tablet_bootstrap.cc:492] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1: Bootstrap starting.
I20260926 13:09:17.984123 5484 tablet_bootstrap.cc:654] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:17.984288 5486 tablet_bootstrap.cc:654] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:17.984455 5484 log.cc:824] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:17.984645 5486 log.cc:824] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:17.984897 5485 tablet_bootstrap.cc:492] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f: Bootstrap starting.
I20260926 13:09:17.985342 5484 tablet_bootstrap.cc:492] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608: No bootstrap required, opened a new log
I20260926 13:09:17.985357 5486 tablet_bootstrap.cc:492] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1: No bootstrap required, opened a new log
I20260926 13:09:17.985412 5486 ts_tablet_manager.cc:1397] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:17.985410 5484 ts_tablet_manager.cc:1397] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:17.985646 5485 tablet_bootstrap.cc:654] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:17.985936 5485 log.cc:824] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:17.986681 5485 tablet_bootstrap.cc:492] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f: No bootstrap required, opened a new log
I20260926 13:09:17.986752 5485 ts_tablet_manager.cc:1397] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 13:09:17.987484 5484 raft_consensus.cc:348] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17bddc45b094c8a985d14574cf0d30f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46721 } } peers { permanent_uuid: "f636bf2010734ab9b86be284af9d43f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34243 } } peers { permanent_uuid: "01d021d42d0f490f800e6ef54f00d608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44459 } }
I20260926 13:09:17.987617 5484 raft_consensus.cc:374] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:17.987648 5484 raft_consensus.cc:729] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d021d42d0f490f800e6ef54f00d608, State: Initialized, Role: FOLLOWER
I20260926 13:09:17.987785 5484 consensus_queue.cc:260] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17bddc45b094c8a985d14574cf0d30f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46721 } } peers { permanent_uuid: "f636bf2010734ab9b86be284af9d43f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34243 } } peers { permanent_uuid: "01d021d42d0f490f800e6ef54f00d608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44459 } }
I20260926 13:09:17.987843 5486 raft_consensus.cc:348] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17bddc45b094c8a985d14574cf0d30f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46721 } } peers { permanent_uuid: "f636bf2010734ab9b86be284af9d43f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34243 } } peers { permanent_uuid: "01d021d42d0f490f800e6ef54f00d608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44459 } }
I20260926 13:09:17.988006 5486 raft_consensus.cc:374] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:17.988042 5486 raft_consensus.cc:729] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f636bf2010734ab9b86be284af9d43f1, State: Initialized, Role: FOLLOWER
I20260926 13:09:17.988040 5484 ts_tablet_manager.cc:1428] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 13:09:17.988157 5208 heartbeater.cc:503] Master 127.0.98.62:38907 was elected leader, sending a full tablet report...
I20260926 13:09:17.988186 5486 consensus_queue.cc:260] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17bddc45b094c8a985d14574cf0d30f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46721 } } peers { permanent_uuid: "f636bf2010734ab9b86be284af9d43f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34243 } } peers { permanent_uuid: "01d021d42d0f490f800e6ef54f00d608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44459 } }
I20260926 13:09:17.988452 5339 heartbeater.cc:503] Master 127.0.98.62:38907 was elected leader, sending a full tablet report...
I20260926 13:09:17.988425 5486 ts_tablet_manager.cc:1428] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:09:17.989435 5485 raft_consensus.cc:348] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17bddc45b094c8a985d14574cf0d30f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46721 } } peers { permanent_uuid: "f636bf2010734ab9b86be284af9d43f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34243 } } peers { permanent_uuid: "01d021d42d0f490f800e6ef54f00d608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44459 } }
I20260926 13:09:17.989574 5485 raft_consensus.cc:374] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:17.989612 5485 raft_consensus.cc:729] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17bddc45b094c8a985d14574cf0d30f, State: Initialized, Role: FOLLOWER
I20260926 13:09:17.989714 5485 consensus_queue.cc:260] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17bddc45b094c8a985d14574cf0d30f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46721 } } peers { permanent_uuid: "f636bf2010734ab9b86be284af9d43f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34243 } } peers { permanent_uuid: "01d021d42d0f490f800e6ef54f00d608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44459 } }
I20260926 13:09:17.989898 5485 ts_tablet_manager.cc:1428] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:09:17.989950 5470 heartbeater.cc:503] Master 127.0.98.62:38907 was elected leader, sending a full tablet report...
I20260926 13:09:18.020269 5490 raft_consensus.cc:482] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:18.020414 5490 raft_consensus.cc:504] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17bddc45b094c8a985d14574cf0d30f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46721 } } peers { permanent_uuid: "f636bf2010734ab9b86be284af9d43f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34243 } } peers { permanent_uuid: "01d021d42d0f490f800e6ef54f00d608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44459 } }
I20260926 13:09:18.020771 5490 leader_election.cc:288] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b17bddc45b094c8a985d14574cf0d30f (127.0.98.3:46721), f636bf2010734ab9b86be284af9d43f1 (127.0.98.2:34243)
I20260926 13:09:18.022235 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.021134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49787 (local address 127.0.98.3:46721)
0926 13:09:18.021190 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:18.021192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:18.021360 (+ 168us) server_negotiation.cc:407] Connection header received
0926 13:09:18.021390 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:18.021391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:18.021397 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:18.021443 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:18.022001 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:18.022002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:18.022003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:18.022007 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:18.022020 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:18.022024 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:18.022082 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:18.022152 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 13:09:18.022167 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 13:09:18.022267 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.021102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60173 (local address 127.0.98.2:34243)
0926 13:09:18.021147 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:18.021149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:18.021157 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:18.021186 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:18.021238 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:18.021243 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:18.021293 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:18.021876 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:18.021877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:18.021878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:18.021883 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:18.021900 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:18.021906 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:18.021960 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:18.022183 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 13:09:18.022207 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:09:18.022249 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.020988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34243 (local address 127.0.98.1:60173)
0926 13:09:18.021045 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:18.021058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:18.021104 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:18.021315 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:18.021316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:18.021318 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:18.021752 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:18.021753 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:18.021761 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:18.021776 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:18.022106 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:18.022107 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:18.022108 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 13:09:18.022149 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 13:09:18.022163 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 13:09:18.022259 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.021101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46721 (local address 127.0.98.1:49787)
0926 13:09:18.021283 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:18.021292 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:18.021346 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:18.021468 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:18.021468 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:18.021471 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:18.021879 (+ 408us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:18.021880 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:18.021887 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:18.021899 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:18.022082 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:18.022082 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:18.022083 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 13:09:18.022131 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 13:09:18.022146 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":115,"threads_started":1}
I20260926 13:09:18.022665 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27977c8902c44a21979b0b74ede0cef6" candidate_uuid: "01d021d42d0f490f800e6ef54f00d608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f636bf2010734ab9b86be284af9d43f1" is_pre_election: true
I20260926 13:09:18.022866 5293 raft_consensus.cc:2454] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d021d42d0f490f800e6ef54f00d608 in term 0.
I20260926 13:09:18.022909 5424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27977c8902c44a21979b0b74ede0cef6" candidate_uuid: "01d021d42d0f490f800e6ef54f00d608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17bddc45b094c8a985d14574cf0d30f" is_pre_election: true
I20260926 13:09:18.023061 5424 raft_consensus.cc:2454] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01d021d42d0f490f800e6ef54f00d608 in term 0.
I20260926 13:09:18.023116 5098 leader_election.cc:302] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d021d42d0f490f800e6ef54f00d608, f636bf2010734ab9b86be284af9d43f1; no voters:
I20260926 13:09:18.023321 5490 raft_consensus.cc:2781] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:18.023396 5490 raft_consensus.cc:482] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:18.023428 5490 raft_consensus.cc:3037] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:18.024171 5490 raft_consensus.cc:504] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17bddc45b094c8a985d14574cf0d30f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46721 } } peers { permanent_uuid: "f636bf2010734ab9b86be284af9d43f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34243 } } peers { permanent_uuid: "01d021d42d0f490f800e6ef54f00d608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44459 } }
I20260926 13:09:18.024314 5490 leader_election.cc:288] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [CANDIDATE]: Term 1 election: Requested vote from peers b17bddc45b094c8a985d14574cf0d30f (127.0.98.3:46721), f636bf2010734ab9b86be284af9d43f1 (127.0.98.2:34243)
I20260926 13:09:18.024480 5424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27977c8902c44a21979b0b74ede0cef6" candidate_uuid: "01d021d42d0f490f800e6ef54f00d608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17bddc45b094c8a985d14574cf0d30f"
I20260926 13:09:18.024528 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27977c8902c44a21979b0b74ede0cef6" candidate_uuid: "01d021d42d0f490f800e6ef54f00d608" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f636bf2010734ab9b86be284af9d43f1"
I20260926 13:09:18.024578 5424 raft_consensus.cc:3037] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:18.024593 5293 raft_consensus.cc:3037] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:18.025473 5293 raft_consensus.cc:2454] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d021d42d0f490f800e6ef54f00d608 in term 1.
I20260926 13:09:18.025473 5424 raft_consensus.cc:2454] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01d021d42d0f490f800e6ef54f00d608 in term 1.
I20260926 13:09:18.025660 5098 leader_election.cc:302] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01d021d42d0f490f800e6ef54f00d608, f636bf2010734ab9b86be284af9d43f1; no voters:
I20260926 13:09:18.025755 5490 raft_consensus.cc:2781] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:18.025897 5490 raft_consensus.cc:686] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [term 1 LEADER]: Becoming Leader. State: Replica: 01d021d42d0f490f800e6ef54f00d608, State: Running, Role: LEADER
I20260926 13:09:18.025975 5490 consensus_queue.cc:237] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17bddc45b094c8a985d14574cf0d30f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46721 } } peers { permanent_uuid: "f636bf2010734ab9b86be284af9d43f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34243 } } peers { permanent_uuid: "01d021d42d0f490f800e6ef54f00d608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44459 } }
I20260926 13:09:18.026823 5019 catalog_manager.cc:5495] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01d021d42d0f490f800e6ef54f00d608 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "01d021d42d0f490f800e6ef54f00d608" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b17bddc45b094c8a985d14574cf0d30f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f636bf2010734ab9b86be284af9d43f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01d021d42d0f490f800e6ef54f00d608" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44459 } health_report { overall_health: HEALTHY } } }
I20260926 13:09:18.038364 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.037380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.0.98.1:44459)
0926 13:09:18.037431 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:18.037433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:18.037441 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:18.037476 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:18.037477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:18.037489 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:18.037542 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:18.038101 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:18.038101 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:18.038102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:18.038105 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:18.038183 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:18.038188 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:18.038240 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:18.038294 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 13:09:18.038306 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 13:09:18.040431 5424 raft_consensus.cc:1261] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f [term 1 FOLLOWER]: Refusing update from remote peer 01d021d42d0f490f800e6ef54f00d608: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:18.040459 5293 raft_consensus.cc:1261] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1 [term 1 FOLLOWER]: Refusing update from remote peer 01d021d42d0f490f800e6ef54f00d608: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:18.040776 5490 consensus_queue.cc:1035] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f636bf2010734ab9b86be284af9d43f1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:18.040920 5494 consensus_queue.cc:1035] T 27977c8902c44a21979b0b74ede0cef6 P 01d021d42d0f490f800e6ef54f00d608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b17bddc45b094c8a985d14574cf0d30f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:18.043459 5497 mvcc.cc:205] Tried to move back new op lower bound from 7333593735330160640 to 7333593735274889216. Current Snapshot: MvccSnapshot[applied={T|T < 7333593735330160640}]
I20260926 13:09:18.043583 5499 mvcc.cc:205] Tried to move back new op lower bound from 7333593735330160640 to 7333593735274889216. Current Snapshot: MvccSnapshot[applied={T|T < 7333593735330160640}]
I20260926 13:09:18.043809 5498 mvcc.cc:205] Tried to move back new op lower bound from 7333593735330160640 to 7333593735274889216. Current Snapshot: MvccSnapshot[applied={T|T < 7333593735330160640}]
I20260926 13:09:18.046280 5019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:18.046372 5019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:18.047145 5019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de30b1c06da647b69c75fbb64a40eae9: Sending DeleteTablet for 3 replicas of tablet 27977c8902c44a21979b0b74ede0cef6
I20260926 13:09:18.047483 5404 tablet_service.cc:1514] Processing DeleteTablet for tablet 27977c8902c44a21979b0b74ede0cef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:18 UTC) from {username='slave'} at 127.0.0.1:47912
I20260926 13:09:18.047535 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 27977c8902c44a21979b0b74ede0cef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:18 UTC) from {username='slave'} at 127.0.0.1:35206
I20260926 13:09:18.047672 5273 tablet_service.cc:1514] Processing DeleteTablet for tablet 27977c8902c44a21979b0b74ede0cef6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:18 UTC) from {username='slave'} at 127.0.0.1:54300
I20260926 13:09:18.047886 5509 tablet_replica.cc:330] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1: stopping tablet replica
I20260926 13:09:18.048002 5509 raft_consensus.cc:2229] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:18.048110 5509 raft_consensus.cc:2258] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:18.048166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 5080
I20260926 13:09:18.048872 5507 tablet_replica.cc:330] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f: stopping tablet replica
I20260926 13:09:18.048985 5507 raft_consensus.cc:2229] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:18.049091 5507 raft_consensus.cc:2258] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:18.049494 5507 ts_tablet_manager.cc:1905] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:18.050932 5509 ts_tablet_manager.cc:1905] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:18.051239 5507 ts_tablet_manager.cc:1918] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:18.051332 5507 log.cc:1196] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/27977c8902c44a21979b0b74ede0cef6
I20260926 13:09:18.051570 5507 ts_tablet_manager.cc:1939] T 27977c8902c44a21979b0b74ede0cef6 P b17bddc45b094c8a985d14574cf0d30f: Deleting consensus metadata
I20260926 13:09:18.052083 5005 catalog_manager.cc:4841] TS b17bddc45b094c8a985d14574cf0d30f (127.0.98.3:46721): tablet 27977c8902c44a21979b0b74ede0cef6 (table test-table [id=0a637f52413b45d9b979558135293d2a]) successfully deleted
I20260926 13:09:18.052479 5509 ts_tablet_manager.cc:1918] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:18.052544 5509 log.cc:1196] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/27977c8902c44a21979b0b74ede0cef6
I20260926 13:09:18.052766 5509 ts_tablet_manager.cc:1939] T 27977c8902c44a21979b0b74ede0cef6 P f636bf2010734ab9b86be284af9d43f1: Deleting consensus metadata
I20260926 13:09:18.053334 5007 catalog_manager.cc:4841] TS f636bf2010734ab9b86be284af9d43f1 (127.0.98.2:34243): tablet 27977c8902c44a21979b0b74ede0cef6 (table test-table [id=0a637f52413b45d9b979558135293d2a]) successfully deleted
W20260926 13:09:18.054158 5005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44459 (error 108)
I20260926 13:09:18.054263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 5211
I20260926 13:09:18.055343 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.055180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44459 (local address 127.0.0.1:35230)
0926 13:09:18.055259 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:18.055306 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 13:09:18.055483 5005 catalog_manager.cc:4584] TS 01d021d42d0f490f800e6ef54f00d608 (127.0.98.1:44459): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27977c8902c44a21979b0b74ede0cef6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44459: connect: Connection refused (error 111)
I20260926 13:09:18.059434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 5342
I20260926 13:09:18.063491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 4989
2026-09-26T13:09:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (601 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 13:09:18.082170 392 mini_oidc.cc:152] Starting JWKS server
I20260926 13:09:18.083369 392 webserver.cc:460] Webserver started at https://127.0.0.1:42789/ using document root <none> and password file <none>
I20260926 13:09:18.083467 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 13:09:18.083635 392 webserver.cc:460] Webserver started at http://127.0.0.1:41957/ using document root <none> and password file <none>
2026-09-26T13:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:18Z Disabled control of system clock
I20260926 13:09:18.091116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44375
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44375
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42789/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/testchainca.cert with env {}
W20260926 13:09:18.172646 5516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:18.172859 5516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:18.172881 5516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:18.174057 5516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:18.174108 5516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:18.174119 5516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:18.174130 5516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:18.174141 5516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:09:18.174152 5516 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:09:18.174171 5516 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42789/jwks.json
I20260926 13:09:18.175859 5516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42789/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:18.176074 5516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:18.176249 5516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:18.180963 5521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:18.180963 5524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:18.181207 5516 server_base.cc:1034] running on GCE node
W20260926 13:09:18.180980 5522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:18.181469 5516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:18.181752 5516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:18.182910 5516 hybrid_clock.cc:648] HybridClock initialized: now 1790428158182894 us; error 35 us; skew 500 ppm
I20260926 13:09:18.193193 5516 webserver.cc:460] Webserver started at http://127.0.98.62:39575/ using document root <none> and password file <none>
I20260926 13:09:18.193471 5516 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:18.193526 5516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:18.193661 5516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:18.194723 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "a9a90a8f4ec3449bbb195db511b1462f"
format_stamp: "Formatted at 2026-09-26 13:09:18 on dist-test-slave-2dw5"
server_key: "0ceb0f83c7352d5d8847001abd9553ab"
server_key_iv: "f8a3cf41c16b2f5b52185a89da4b4108"
server_key_version: "encryptionkey@0"
I20260926 13:09:18.195098 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "a9a90a8f4ec3449bbb195db511b1462f"
format_stamp: "Formatted at 2026-09-26 13:09:18 on dist-test-slave-2dw5"
server_key: "0ceb0f83c7352d5d8847001abd9553ab"
server_key_iv: "f8a3cf41c16b2f5b52185a89da4b4108"
server_key_version: "encryptionkey@0"
I20260926 13:09:18.196424 5516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:18.197227 5530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:18.197424 5516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:18.197517 5516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "a9a90a8f4ec3449bbb195db511b1462f"
format_stamp: "Formatted at 2026-09-26 13:09:18 on dist-test-slave-2dw5"
server_key: "0ceb0f83c7352d5d8847001abd9553ab"
server_key_iv: "f8a3cf41c16b2f5b52185a89da4b4108"
server_key_version: "encryptionkey@0"
I20260926 13:09:18.197588 5516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:18.207106 5516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:18.207405 5516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:18.207531 5516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:18.211088 5516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44375
I20260926 13:09:18.211122 5582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44375 every 8 connection(s)
I20260926 13:09:18.211462 5516 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:18.212381 5583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:18.214635 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f: Bootstrap starting.
I20260926 13:09:18.215247 5583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:18.215507 5583 log.cc:824] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:18.215880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 5516
I20260926 13:09:18.215969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:18.216125 392 external_mini_cluster.cc:1428] Setting key 26c125a9ed1f0777a26d2a3097bf7981
I20260926 13:09:18.216171 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f: No bootstrap required, opened a new log
I20260926 13:09:18.218943 5583 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a90a8f4ec3449bbb195db511b1462f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44375 } }
I20260926 13:09:18.219113 5583 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:18.219141 5583 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a90a8f4ec3449bbb195db511b1462f, State: Initialized, Role: FOLLOWER
I20260926 13:09:18.219259 5583 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a90a8f4ec3449bbb195db511b1462f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44375 } }
I20260926 13:09:18.219321 5583 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:18.219347 5583 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:18.219390 5583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:18.219928 5583 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a90a8f4ec3449bbb195db511b1462f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44375 } }
I20260926 13:09:18.220014 5583 leader_election.cc:302] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9a90a8f4ec3449bbb195db511b1462f; no voters:
I20260926 13:09:18.220165 5583 leader_election.cc:288] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:18.220225 5588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:18.220386 5588 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [term 1 LEADER]: Becoming Leader. State: Replica: a9a90a8f4ec3449bbb195db511b1462f, State: Running, Role: LEADER
I20260926 13:09:18.220430 5583 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:18.220468 5588 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a90a8f4ec3449bbb195db511b1462f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44375 } }
I20260926 13:09:18.220849 5588 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9a90a8f4ec3449bbb195db511b1462f. Latest consensus state: current_term: 1 leader_uuid: "a9a90a8f4ec3449bbb195db511b1462f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a90a8f4ec3449bbb195db511b1462f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44375 } } }
I20260926 13:09:18.220924 5588 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:18.221278 5597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:18.220911 5589 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9a90a8f4ec3449bbb195db511b1462f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9a90a8f4ec3449bbb195db511b1462f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44375 } } }
I20260926 13:09:18.221541 5589 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:18.221918 5597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:18.222656 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.216591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.0.98.62:44375)
0926 13:09:18.216811 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:18.216820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:18.216855 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:09:18.217529 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:18.217545 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:18.217630 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:18.217785 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:18.217980 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.219372 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.220171 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.220873 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.221778 (+ 905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:18.221949 (+ 171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:18.221953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:18.221953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:18.222151 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:18.222190 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:18.222204 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:18.222281 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:18.222381 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 13:09:18.222459 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260926 13:09:18.223544 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:09:18.223768 5597 catalog_manager.cc:1347] Generated new cluster ID: 73989829a1354ea5b89ec45e0b2ad035
I20260926 13:09:18.223819 5597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:18.264365 5597 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:18.264909 5597 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:18.273208 5597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9a90a8f4ec3449bbb195db511b1462f: Generated new TSK 0
I20260926 13:09:18.273439 5597 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 13:09:18.306370 5613 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 13:09:18.306483 5613 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 13:09:18.306870 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.282525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.0.98.62:44375)
0926 13:09:18.282565 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:18.282567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:18.282586 (+ 19us) server_negotiation.cc:407] Connection header received
0926 13:09:18.282632 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:18.282633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:18.282646 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:18.282727 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:09:18.282992 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.283756 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.285085 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.285555 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.285609 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:18.285933 (+ 324us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:09:18.306423 (+ 20490us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 13:09:18.306774 (+ 351us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38972: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 13:09:18.307149 5587 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38972: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 13:09:18.307210 5612 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:18.282183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44375 (local address 127.0.0.1:38972)
0926 13:09:18.282376 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:18.282390 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:18.282466 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:18.282746 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:18.282747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:18.282750 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:09:18.282857 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:18.282860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.284033 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.284034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:18.284970 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:18.284974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.285683 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.285687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:18.285753 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:18.285784 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:09:18.306717 (+ 20933us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 13:09:18.306876 (+ 159us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44375: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:44375: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42789/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 13:09:18.307703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 5516
2026-09-26T13:09:18Z chronyd exiting
I20260926 13:09:18.345654 392 test_util.cc:175] -----------------------------------------------
I20260926 13:09:18.345736 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428136114168-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (264 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 13:09:18.346606 392 mini_oidc.cc:152] Starting JWKS server
I20260926 13:09:18.347318 392 webserver.cc:460] Webserver started at https://127.0.0.1:33893/ using document root <none> and password file <none>
I20260926 13:09:18.347363 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 13:09:18.347491 392 webserver.cc:460] Webserver started at http://127.0.0.1:46271/ using document root <none> and password file <none>
2026-09-26T13:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:18Z Disabled control of system clock
I20260926 13:09:18.354647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39431
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39431
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33893/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/testchainca.cert with env {}
W20260926 13:09:18.434530 5619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:18.434753 5619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:18.434785 5619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:18.436029 5619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:18.436079 5619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:18.436095 5619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:18.436111 5619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:18.436127 5619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:09:18.436144 5619 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:09:18.436167 5619 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33893/jwks.json
I20260926 13:09:18.437757 5619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33893/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:18.438009 5619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:18.438205 5619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:18.443133 5624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:18.443133 5627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:18.443127 5625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:18.443455 5619 server_base.cc:1034] running on GCE node
I20260926 13:09:18.443691 5619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:18.443961 5619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:18.445093 5619 hybrid_clock.cc:648] HybridClock initialized: now 1790428158445066 us; error 49 us; skew 500 ppm
I20260926 13:09:18.450316 5619 webserver.cc:460] Webserver started at http://127.0.98.62:39173/ using document root <none> and password file <none>
I20260926 13:09:18.450537 5619 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:18.450580 5619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:18.450680 5619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:18.451846 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "4f0477351d7d49789d1d189df6edb145"
format_stamp: "Formatted at 2026-09-26 13:09:18 on dist-test-slave-2dw5"
server_key: "97fbdbaaa9187c017eb2c8db732b7293"
server_key_iv: "f8786cfd4b154b60e81ebf341f38df01"
server_key_version: "encryptionkey@0"
I20260926 13:09:18.452160 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f0477351d7d49789d1d189df6edb145"
format_stamp: "Formatted at 2026-09-26 13:09:18 on dist-test-slave-2dw5"
server_key: "97fbdbaaa9187c017eb2c8db732b7293"
server_key_iv: "f8786cfd4b154b60e81ebf341f38df01"
server_key_version: "encryptionkey@0"
I20260926 13:09:18.453541 5619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:18.454320 5633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:18.454535 5619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:18.454629 5619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "4f0477351d7d49789d1d189df6edb145"
format_stamp: "Formatted at 2026-09-26 13:09:18 on dist-test-slave-2dw5"
server_key: "97fbdbaaa9187c017eb2c8db732b7293"
server_key_iv: "f8786cfd4b154b60e81ebf341f38df01"
server_key_version: "encryptionkey@0"
I20260926 13:09:18.454696 5619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:18.465678 5619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:18.465982 5619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:18.466106 5619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:18.470041 5619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39431
I20260926 13:09:18.470084 5685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39431 every 8 connection(s)
I20260926 13:09:18.470424 5619 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:18.471410 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:18.473793 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145: Bootstrap starting.
I20260926 13:09:18.474337 5686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:18.474565 5686 log.cc:824] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:18.475405 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145: No bootstrap required, opened a new log
I20260926 13:09:18.476971 5686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0477351d7d49789d1d189df6edb145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39431 } }
I20260926 13:09:18.477118 5686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:18.477147 5686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f0477351d7d49789d1d189df6edb145, State: Initialized, Role: FOLLOWER
I20260926 13:09:18.477257 5686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0477351d7d49789d1d189df6edb145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39431 } }
I20260926 13:09:18.477309 5686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:18.477329 5686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:18.477383 5686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:18.477900 5686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0477351d7d49789d1d189df6edb145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39431 } }
I20260926 13:09:18.477988 5686 leader_election.cc:302] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f0477351d7d49789d1d189df6edb145; no voters:
I20260926 13:09:18.478152 5686 leader_election.cc:288] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:18.478193 5689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:18.478376 5689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [term 1 LEADER]: Becoming Leader. State: Replica: 4f0477351d7d49789d1d189df6edb145, State: Running, Role: LEADER
I20260926 13:09:18.478406 5686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:18.478485 5689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0477351d7d49789d1d189df6edb145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39431 } }
I20260926 13:09:18.478756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 5619
I20260926 13:09:18.478886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:18.479010 5691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f0477351d7d49789d1d189df6edb145. Latest consensus state: current_term: 1 leader_uuid: "4f0477351d7d49789d1d189df6edb145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0477351d7d49789d1d189df6edb145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39431 } } }
I20260926 13:09:18.479074 392 external_mini_cluster.cc:1428] Setting key bdd1f1808332562b5498e2f1590158b9
I20260926 13:09:18.479094 5691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:18.479692 5690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f0477351d7d49789d1d189df6edb145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0477351d7d49789d1d189df6edb145" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39431 } } }
I20260926 13:09:18.479771 5690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:18.480415 5698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:18.480965 5698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:18.482646 5698 catalog_manager.cc:1347] Generated new cluster ID: d8b413532ca948c0b16f75f3d024355f
I20260926 13:09:18.482703 5698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:18.485260 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.479917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.0.98.62:39431)
0926 13:09:18.480103 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:18.480113 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:18.480148 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:09:18.480287 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:18.480295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:18.480368 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:18.480520 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:18.480739 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.482178 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.483283 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.483778 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.483899 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:18.484592 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:18.484597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:18.484597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:18.484778 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:18.484815 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:18.484827 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:18.484888 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:18.484975 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 13:09:18.485056 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260926 13:09:18.486174 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:09:18.490074 5698 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:18.490698 5698 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:18.500733 5698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f0477351d7d49789d1d189df6edb145: Generated new TSK 0
I20260926 13:09:18.500926 5698 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 13:09:18.523102 5716 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 13:09:18.523200 5716 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 13:09:18.523561 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.508716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.0.98.62:39431)
0926 13:09:18.508763 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:18.508765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:18.508851 (+ 86us) server_negotiation.cc:407] Connection header received
0926 13:09:18.508939 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:18.508940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:18.508954 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:18.509021 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:09:18.509257 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.509794 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.510766 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.511053 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.511119 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:18.511350 (+ 231us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:09:18.523325 (+ 11975us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 13:09:18.523489 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50574: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 13:09:18.523622 5701 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50574: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 13:09:18.523649 5715 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:18.508627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39431 (local address 127.0.0.1:50574)
0926 13:09:18.508772 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:18.508784 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:18.508857 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:18.509114 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:18.509116 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:18.509117 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:09:18.509181 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:18.509184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.509872 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.509873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:18.510658 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:18.510662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.511180 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.511184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:18.511234 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:18.511240 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:09:18.523447 (+ 12207us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 13:09:18.523553 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260926 13:09:18.523947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 5619
2026-09-26T13:09:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (214 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 13:09:18.561062 392 mini_oidc.cc:152] Starting JWKS server
I20260926 13:09:18.561791 392 webserver.cc:460] Webserver started at https://127.0.0.1:37477/ using document root <none> and password file <none>
I20260926 13:09:18.561836 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 13:09:18.561960 392 webserver.cc:460] Webserver started at http://127.0.0.1:43519/ using document root <none> and password file <none>
2026-09-26T13:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:18Z Disabled control of system clock
I20260926 13:09:18.569640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46423
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46423
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37477/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 13:09:18.649039 5722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:18.649272 5722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:18.649302 5722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:18.650600 5722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:18.650652 5722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:18.650668 5722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:18.650686 5722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:18.650701 5722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:09:18.650719 5722 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:09:18.650736 5722 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37477/jwks.json
I20260926 13:09:18.652194 5722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46423
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37477/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:18.652464 5722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:18.652740 5722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:18.657573 5727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:18.657568 5730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:18.657572 5728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:18.657934 5722 server_base.cc:1034] running on GCE node
I20260926 13:09:18.658192 5722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:18.658473 5722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:18.659655 5722 hybrid_clock.cc:648] HybridClock initialized: now 1790428158659636 us; error 34 us; skew 500 ppm
I20260926 13:09:18.660872 5722 webserver.cc:460] Webserver started at http://127.0.98.62:39333/ using document root <none> and password file <none>
I20260926 13:09:18.661087 5722 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:18.661140 5722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:18.661269 5722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:18.662320 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "bda64d1c01fb4e45a0336331e3129013"
format_stamp: "Formatted at 2026-09-26 13:09:18 on dist-test-slave-2dw5"
server_key: "09f88aa853d1caabbf944a94bedf379d"
server_key_iv: "407db5ce2a6548f1601cb859b1c40e32"
server_key_version: "encryptionkey@0"
I20260926 13:09:18.662643 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "bda64d1c01fb4e45a0336331e3129013"
format_stamp: "Formatted at 2026-09-26 13:09:18 on dist-test-slave-2dw5"
server_key: "09f88aa853d1caabbf944a94bedf379d"
server_key_iv: "407db5ce2a6548f1601cb859b1c40e32"
server_key_version: "encryptionkey@0"
I20260926 13:09:18.663935 5722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:18.664636 5736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:18.664847 5722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:18.664932 5722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "bda64d1c01fb4e45a0336331e3129013"
format_stamp: "Formatted at 2026-09-26 13:09:18 on dist-test-slave-2dw5"
server_key: "09f88aa853d1caabbf944a94bedf379d"
server_key_iv: "407db5ce2a6548f1601cb859b1c40e32"
server_key_version: "encryptionkey@0"
I20260926 13:09:18.664994 5722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:18.681872 5722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:18.682211 5722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:18.682361 5722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:18.686585 5722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46423
I20260926 13:09:18.686627 5788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46423 every 8 connection(s)
I20260926 13:09:18.687021 5722 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:18.687741 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:18.690702 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013: Bootstrap starting.
I20260926 13:09:18.691356 5789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:18.691633 5789 log.cc:824] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:18.692332 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013: No bootstrap required, opened a new log
I20260926 13:09:18.693998 5789 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda64d1c01fb4e45a0336331e3129013" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260926 13:09:18.694157 5789 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:18.694182 5789 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda64d1c01fb4e45a0336331e3129013, State: Initialized, Role: FOLLOWER
I20260926 13:09:18.694334 5789 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda64d1c01fb4e45a0336331e3129013" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260926 13:09:18.694393 5789 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:18.694414 5789 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:18.694458 5789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:18.694492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 5722
I20260926 13:09:18.694593 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:18.694749 392 external_mini_cluster.cc:1428] Setting key 23d2a08279fbe08195be60be94f51db7
I20260926 13:09:18.695114 5789 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda64d1c01fb4e45a0336331e3129013" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260926 13:09:18.695232 5789 leader_election.cc:302] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda64d1c01fb4e45a0336331e3129013; no voters:
I20260926 13:09:18.695403 5789 leader_election.cc:288] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:18.695519 5794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:18.695657 5789 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:18.695662 5794 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [term 1 LEADER]: Becoming Leader. State: Replica: bda64d1c01fb4e45a0336331e3129013, State: Running, Role: LEADER
I20260926 13:09:18.695767 5794 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda64d1c01fb4e45a0336331e3129013" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } }
I20260926 13:09:18.696245 5795 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda64d1c01fb4e45a0336331e3129013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda64d1c01fb4e45a0336331e3129013" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } } }
I20260926 13:09:18.696322 5795 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:18.696754 5803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:18.697067 5796 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda64d1c01fb4e45a0336331e3129013. Latest consensus state: current_term: 1 leader_uuid: "bda64d1c01fb4e45a0336331e3129013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda64d1c01fb4e45a0336331e3129013" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46423 } } }
I20260926 13:09:18.697173 5796 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:18.697371 5803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:18.698879 5803 catalog_manager.cc:1347] Generated new cluster ID: 5096f0c3b826464bb7b17638e1332b31
I20260926 13:09:18.698926 5803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:18.700742 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.695249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.0.98.62:46423)
0926 13:09:18.695501 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:18.695511 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:18.695549 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:09:18.695675 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:18.695684 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:18.695763 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:18.695905 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:18.696216 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.697678 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.698633 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.699232 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.699349 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:18.700038 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:18.700042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:18.700042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:18.700255 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:18.700299 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:18.700315 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:18.700398 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:18.700435 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 13:09:18.700515 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
I20260926 13:09:18.701659 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:09:18.719247 5803 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:18.720000 5803 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:18.728524 5803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda64d1c01fb4e45a0336331e3129013: Generated new TSK 0
I20260926 13:09:18.728719 5803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:09:18.741668 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.737926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54466 (local address 127.0.98.62:46423)
0926 13:09:18.737974 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:18.737977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:18.738040 (+ 63us) server_negotiation.cc:407] Connection header received
0926 13:09:18.738091 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:18.738092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:18.738104 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:18.738186 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:09:18.738356 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.739179 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.739962 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.740276 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.740347 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:18.740462 (+ 115us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:09:18.741443 (+ 981us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 13:09:18.741583 (+ 140us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 13:09:18.741732 5793 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 13:09:18.743615 5818 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:18.737844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46423 (local address 127.0.0.1:54466)
0926 13:09:18.737971 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:18.737982 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:18.738037 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:18.738244 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:18.738245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:18.738247 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:09:18.738297 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:18.738299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.739275 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.739276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:18.739888 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:18.739890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.740341 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.740345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:18.740380 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:18.740385 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:09:18.741551 (+ 1166us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 13:09:18.743515 (+ 1964us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46423: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37477/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":1783,"thread_start_us":72,"threads_started":1}
I20260926 13:09:18.743907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 5722
E20260926 13:09:18.752871 5819 webserver.cc:533] Webserver: sslize failed:
E20260926 13:09:18.752969 5819 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T13:09:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T13:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:18Z Disabled control of system clock
I20260926 13:09:18.786859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43509
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43509
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/testchainca.cert with env {}
W20260926 13:09:18.869155 5823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:18.869375 5823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:18.869400 5823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:18.870551 5823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:18.870594 5823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:18.870604 5823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:18.870617 5823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:18.870630 5823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:09:18.870640 5823 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:09:18.870649 5823 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 13:09:18.872133 5823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:18.872339 5823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:18.872517 5823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:18.877468 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:18.877432 5823 server_base.cc:1034] running on GCE node
W20260926 13:09:18.877452 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:18.877481 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:18.877905 5823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:18.878185 5823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:18.879340 5823 hybrid_clock.cc:648] HybridClock initialized: now 1790428158879320 us; error 36 us; skew 500 ppm
I20260926 13:09:18.880707 5823 webserver.cc:460] Webserver started at http://127.0.98.62:36429/ using document root <none> and password file <none>
I20260926 13:09:18.880923 5823 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:18.880972 5823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:18.881093 5823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:18.882266 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "d2e290f646de4d5e8e19ecd033093259"
format_stamp: "Formatted at 2026-09-26 13:09:18 on dist-test-slave-2dw5"
server_key: "2b73d0090de2c5a481c35f99711e921e"
server_key_iv: "2b2642a39c991b3fb23bfe3090820e9e"
server_key_version: "encryptionkey@0"
I20260926 13:09:18.882596 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "d2e290f646de4d5e8e19ecd033093259"
format_stamp: "Formatted at 2026-09-26 13:09:18 on dist-test-slave-2dw5"
server_key: "2b73d0090de2c5a481c35f99711e921e"
server_key_iv: "2b2642a39c991b3fb23bfe3090820e9e"
server_key_version: "encryptionkey@0"
I20260926 13:09:18.883919 5823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:18.884786 5837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:18.884956 5823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:18.885039 5823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "d2e290f646de4d5e8e19ecd033093259"
format_stamp: "Formatted at 2026-09-26 13:09:18 on dist-test-slave-2dw5"
server_key: "2b73d0090de2c5a481c35f99711e921e"
server_key_iv: "2b2642a39c991b3fb23bfe3090820e9e"
server_key_version: "encryptionkey@0"
I20260926 13:09:18.885106 5823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:18.897035 5823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:18.897389 5823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:18.897543 5823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:18.901913 5823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43509
I20260926 13:09:18.902002 5889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43509 every 8 connection(s)
I20260926 13:09:18.902345 5823 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:18.903344 5890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:18.905665 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259: Bootstrap starting.
I20260926 13:09:18.906246 5890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:18.906491 5890 log.cc:824] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:18.907260 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259: No bootstrap required, opened a new log
I20260926 13:09:18.908921 5890 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e290f646de4d5e8e19ecd033093259" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43509 } }
I20260926 13:09:18.909068 5890 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:18.909092 5890 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e290f646de4d5e8e19ecd033093259, State: Initialized, Role: FOLLOWER
I20260926 13:09:18.909220 5890 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e290f646de4d5e8e19ecd033093259" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43509 } }
I20260926 13:09:18.909281 5890 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:18.909301 5890 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:18.909327 5890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:18.909859 5890 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e290f646de4d5e8e19ecd033093259" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43509 } }
I20260926 13:09:18.909948 5890 leader_election.cc:302] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2e290f646de4d5e8e19ecd033093259; no voters:
I20260926 13:09:18.910145 5890 leader_election.cc:288] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:18.910199 5893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:18.910363 5893 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [term 1 LEADER]: Becoming Leader. State: Replica: d2e290f646de4d5e8e19ecd033093259, State: Running, Role: LEADER
I20260926 13:09:18.910406 5890 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:18.910470 5893 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e290f646de4d5e8e19ecd033093259" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43509 } }
I20260926 13:09:18.910812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 5823
I20260926 13:09:18.910933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:18.911123 392 external_mini_cluster.cc:1428] Setting key 0159fa2327c8ef8eabe975b35b34b834
I20260926 13:09:18.911198 5894 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2e290f646de4d5e8e19ecd033093259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e290f646de4d5e8e19ecd033093259" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43509 } } }
I20260926 13:09:18.911275 5894 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:18.911679 5902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:18.912191 5902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:18.912451 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2e290f646de4d5e8e19ecd033093259. Latest consensus state: current_term: 1 leader_uuid: "d2e290f646de4d5e8e19ecd033093259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e290f646de4d5e8e19ecd033093259" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43509 } } }
I20260926 13:09:18.912554 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:18.913916 5902 catalog_manager.cc:1347] Generated new cluster ID: 14f87e2983a84431b72f3d2e01d3775b
I20260926 13:09:18.913965 5902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:18.926826 5902 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:18.927862 5902 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:18.930275 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.912585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.0.98.62:43509)
0926 13:09:18.912777 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:18.912786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:18.913377 (+ 591us) server_negotiation.cc:407] Connection header received
0926 13:09:18.913513 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:18.913521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:18.913591 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:18.913722 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:18.926318 (+ 12596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.927526 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.928420 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.928854 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.928971 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:18.929610 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:18.929614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:18.929615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:18.929807 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:18.929841 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:18.929854 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:18.929908 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:18.929983 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 13:09:18.930079 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260926 13:09:18.931114 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:09:18.935807 5902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2e290f646de4d5e8e19ecd033093259: Generated new TSK 0
I20260926 13:09:18.936000 5902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:09:19.405092 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:18.941012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.0.98.62:43509)
0926 13:09:18.941061 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:18.941063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:18.941143 (+ 80us) server_negotiation.cc:407] Connection header received
0926 13:09:18.941183 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:18.941184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:18.941197 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:18.941285 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:09:18.941391 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.942170 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.943033 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.943355 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.943432 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:18.943586 (+ 154us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:09:19.404855 (+461269us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 13:09:19.405034 (+ 179us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32888: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 13:09:19.405243 5912 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32888: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 13:09:19.405359 5919 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:18.940940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43509 (local address 127.0.0.1:32888)
0926 13:09:18.941065 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:18.941074 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:18.941134 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:18.941279 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:18.941280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:18.941282 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:09:18.941334 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:18.941336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.942274 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.942276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:18.942956 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:18.942959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:18.943435 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:18.943439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:18.943482 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:18.943487 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:09:19.405011 (+461524us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 13:09:19.405222 (+ 211us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43509: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260926 13:09:19.405686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 5823
2026-09-26T13:09:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (642 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:19 dist-test-slave-2dw5 krb5kdc[5931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:19 dist-test-slave-2dw5 krb5kdc[5931](info): set up 2 sockets
Sep 26 13:09:19 dist-test-slave-2dw5 krb5kdc[5931](info): commencing operation
krb5kdc: starting...
W20260926 13:09:20.566829 392 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:20 dist-test-slave-2dw5 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:09:20.685554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33453
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:20.764999 5944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:20.765239 5944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:20.765260 5944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:20.766443 5944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:20.766490 5944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:20.766505 5944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:20.766522 5944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:20.766533 5944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:20.768124 5944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:20.768347 5944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:20.768544 5944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:20.773924 5950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:20.773983 5952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:20.773983 5949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:20.774196 5944 server_base.cc:1034] running on GCE node
I20260926 13:09:20.774458 5944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:20.774746 5944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:20.775889 5944 hybrid_clock.cc:648] HybridClock initialized: now 1790428160775868 us; error 32 us; skew 500 ppm
Sep 26 13:09:20 dist-test-slave-2dw5 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:20.778256 5944 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:09:20.778658 5944 webserver.cc:460] Webserver started at http://127.0.98.62:38017/ using document root <none> and password file <none>
I20260926 13:09:20.778867 5944 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:20.778921 5944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:20.779017 5944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:20.780050 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "1ccb4be7a64b471d9de17f398b410bbe"
format_stamp: "Formatted at 2026-09-26 13:09:20 on dist-test-slave-2dw5"
server_key: "8134e735a3b1a4231db89b1886566506"
server_key_iv: "2f91960fa6e735e2baa2e399418d965d"
server_key_version: "encryptionkey@0"
I20260926 13:09:20.780339 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "1ccb4be7a64b471d9de17f398b410bbe"
format_stamp: "Formatted at 2026-09-26 13:09:20 on dist-test-slave-2dw5"
server_key: "8134e735a3b1a4231db89b1886566506"
server_key_iv: "2f91960fa6e735e2baa2e399418d965d"
server_key_version: "encryptionkey@0"
I20260926 13:09:20.781586 5944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:20.782248 5959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:20.782406 5944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:20.782518 5944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "1ccb4be7a64b471d9de17f398b410bbe"
format_stamp: "Formatted at 2026-09-26 13:09:20 on dist-test-slave-2dw5"
server_key: "8134e735a3b1a4231db89b1886566506"
server_key_iv: "2f91960fa6e735e2baa2e399418d965d"
server_key_version: "encryptionkey@0"
I20260926 13:09:20.782586 5944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:20.805343 5944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:20.807317 5944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:20.807452 5944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:20.811334 5944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33453
I20260926 13:09:20.811365 6011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33453 every 8 connection(s)
I20260926 13:09:20.811837 5944 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:20.812443 6012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:20.814713 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe: Bootstrap starting.
I20260926 13:09:20.815302 6012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:20.815560 6012 log.cc:824] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:20.816246 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe: No bootstrap required, opened a new log
I20260926 13:09:20.817795 6012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ccb4be7a64b471d9de17f398b410bbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } }
I20260926 13:09:20.817942 6012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:20.817965 6012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ccb4be7a64b471d9de17f398b410bbe, State: Initialized, Role: FOLLOWER
I20260926 13:09:20.818086 6012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ccb4be7a64b471d9de17f398b410bbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } }
I20260926 13:09:20.818156 6012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:20.818176 6012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:20.818205 6012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:20.819115 6012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ccb4be7a64b471d9de17f398b410bbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } }
I20260926 13:09:20.819207 6012 leader_election.cc:302] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ccb4be7a64b471d9de17f398b410bbe; no voters:
I20260926 13:09:20.819360 6012 leader_election.cc:288] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:20.819412 6015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:20.819563 6015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [term 1 LEADER]: Becoming Leader. State: Replica: 1ccb4be7a64b471d9de17f398b410bbe, State: Running, Role: LEADER
I20260926 13:09:20.819610 6012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:20.819693 6015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ccb4be7a64b471d9de17f398b410bbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } }
I20260926 13:09:20.820022 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ccb4be7a64b471d9de17f398b410bbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ccb4be7a64b471d9de17f398b410bbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } } }
I20260926 13:09:20.820051 6017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ccb4be7a64b471d9de17f398b410bbe. Latest consensus state: current_term: 1 leader_uuid: "1ccb4be7a64b471d9de17f398b410bbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ccb4be7a64b471d9de17f398b410bbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33453 } } }
I20260926 13:09:20.820106 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:20.820111 6017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:20.820478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 5944
I20260926 13:09:20.820580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:20.820770 392 external_mini_cluster.cc:1428] Setting key ab1ecd1f899b8e093792b132ac7c4f2c
I20260926 13:09:20.820941 6023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:20.821746 6023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:20.823395 6023 catalog_manager.cc:1347] Generated new cluster ID: c600e15aeb5a4cf7a9c9780327fb7831
I20260926 13:09:20.823449 6023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:09:20 dist-test-slave-2dw5 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:20.828925 6023 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:20.829674 6023 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:20.830540 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:20.821221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.0.98.62:33453)
0926 13:09:20.821449 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:20.821458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:20.821485 (+ 27us) server_negotiation.cc:407] Connection header received
0926 13:09:20.821586 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:20.821591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:20.821648 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:20.821783 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:20.822459 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:20.823564 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:20.824271 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:20.824681 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:20.824772 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:20.826925 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:20.826929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:20.826929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:20.826940 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:20.829053 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:20.829358 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:20.829361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:20.829362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:20.829445 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:20.829615 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:20.829616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:20.829617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:20.829871 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:20.830033 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:20.830126 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 13:09:20.830365 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":64,"threads_started":1}
I20260926 13:09:20.833806 6023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ccb4be7a64b471d9de17f398b410bbe: Generated new TSK 0
I20260926 13:09:20.833990 6023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:09:20.865283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33453
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:20.947799 6038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:20.948027 6038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:20.948060 6038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:20.949225 6038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:20.949270 6038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:20.949306 6038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:20.950877 6038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:20.951117 6038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:20.951323 6038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:20.956334 6046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:20.956357 6043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:20.956517 6044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:20.956489 6038 server_base.cc:1034] running on GCE node
I20260926 13:09:20.956753 6038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:20.957015 6038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:20.958173 6038 hybrid_clock.cc:648] HybridClock initialized: now 1790428160958152 us; error 30 us; skew 500 ppm
Sep 26 13:09:20 dist-test-slave-2dw5 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:20.960660 6038 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:09:20.961081 6038 webserver.cc:460] Webserver started at http://127.0.98.1:37059/ using document root <none> and password file <none>
I20260926 13:09:20.961274 6038 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:20.961320 6038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:20.961440 6038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:20.962495 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "ae5f1435500b4251b2730020f8da40f4"
format_stamp: "Formatted at 2026-09-26 13:09:20 on dist-test-slave-2dw5"
server_key: "85be08bbde58a0aad51c4712a5b20e7b"
server_key_iv: "67dfda228d19fc1c392c59e031c5365d"
server_key_version: "encryptionkey@0"
I20260926 13:09:20.962813 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ae5f1435500b4251b2730020f8da40f4"
format_stamp: "Formatted at 2026-09-26 13:09:20 on dist-test-slave-2dw5"
server_key: "85be08bbde58a0aad51c4712a5b20e7b"
server_key_iv: "67dfda228d19fc1c392c59e031c5365d"
server_key_version: "encryptionkey@0"
I20260926 13:09:20.964195 6038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:20.964929 6053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:20.965095 6038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:20.965176 6038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "ae5f1435500b4251b2730020f8da40f4"
format_stamp: "Formatted at 2026-09-26 13:09:20 on dist-test-slave-2dw5"
server_key: "85be08bbde58a0aad51c4712a5b20e7b"
server_key_iv: "67dfda228d19fc1c392c59e031c5365d"
server_key_version: "encryptionkey@0"
I20260926 13:09:20.965240 6038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:20.975841 6038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:20.977591 6038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:20.977720 6038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:20.977922 6038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:20.978214 6038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:20.978246 6038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:20.978273 6038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:20.978296 6038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:20.984021 6038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37917
I20260926 13:09:20.984071 6166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37917 every 8 connection(s)
I20260926 13:09:20.984376 6038 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:09:20 dist-test-slave-2dw5 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:20.990777 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 6038
I20260926 13:09:20.990895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:20.991087 392 external_mini_cluster.cc:1428] Setting key af942291f4728a80ff366d388f982451
I20260926 13:09:20.993481 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:20.985071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45185 (local address 127.0.98.62:33453)
0926 13:09:20.985142 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:20.985144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:20.985345 (+ 201us) server_negotiation.cc:407] Connection header received
0926 13:09:20.986325 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:20.986326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:20.986342 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:20.986394 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:20.987044 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:20.987547 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:20.988518 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:20.988715 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:20.988770 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:20.990744 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:20.990746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:20.990746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:20.990755 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:20.992132 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:20.992500 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:20.992501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:20.992502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:20.992565 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:20.992846 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:20.992847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:20.992847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:20.992975 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:20.993042 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:20.993344 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 13:09:20.993417 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:09:20.993731 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:20.985075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33453 (local address 127.0.98.1:45185)
0926 13:09:20.985226 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:20.985262 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:20.986218 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:20.986473 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:20.986478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:20.986732 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:20.986954 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:20.986959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:20.987629 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:20.987630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:20.988433 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:20.988435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:20.988784 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:20.988785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:20.988899 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:20.989340 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:20.989348 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:20.990643 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:20.992205 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:20.992213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:20.992215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:20.992435 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:20.992621 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:20.992622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:20.992622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:20.992756 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:20.993043 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:20.993049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:20.993216 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 13:09:20.993327 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 13:09:20.993499 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260926 13:09:20.994319 6167 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33453
I20260926 13:09:20.994414 6167 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:20.994656 6167 heartbeater.cc:511] Master 127.0.98.62:33453 requested a full tablet report, sending...
I20260926 13:09:20.995280 5976 ts_manager.cc:194] Registered new tserver with Master: ae5f1435500b4251b2730020f8da40f4 (127.0.98.1:37917)
I20260926 13:09:20.996196 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45185
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:09:21.018728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33453
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:21.100301 6172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:21.100517 6172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:21.100540 6172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:21.101646 6172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:21.101681 6172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:21.101706 6172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:21.103232 6172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:21.103547 6172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:21.103744 6172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:21.108914 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:21.108919 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:21.109120 6172 server_base.cc:1034] running on GCE node
W20260926 13:09:21.108916 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:21.109380 6172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:21.109591 6172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:21.110749 6172 hybrid_clock.cc:648] HybridClock initialized: now 1790428161110733 us; error 28 us; skew 500 ppm
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:21.113116 6172 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:09:21.113579 6172 webserver.cc:460] Webserver started at http://127.0.98.2:35827/ using document root <none> and password file <none>
I20260926 13:09:21.113772 6172 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:21.113817 6172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:21.113940 6172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:21.115042 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "9908c3cf172a47a484c792d02e268217"
format_stamp: "Formatted at 2026-09-26 13:09:21 on dist-test-slave-2dw5"
server_key: "730ed45ba79b6142b5e12a06adbfde7c"
server_key_iv: "c12f68f271be00b81cb8a71dbce291f9"
server_key_version: "encryptionkey@0"
I20260926 13:09:21.115376 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9908c3cf172a47a484c792d02e268217"
format_stamp: "Formatted at 2026-09-26 13:09:21 on dist-test-slave-2dw5"
server_key: "730ed45ba79b6142b5e12a06adbfde7c"
server_key_iv: "c12f68f271be00b81cb8a71dbce291f9"
server_key_version: "encryptionkey@0"
I20260926 13:09:21.116626 6172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:21.117260 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:21.117412 6172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:21.117491 6172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "9908c3cf172a47a484c792d02e268217"
format_stamp: "Formatted at 2026-09-26 13:09:21 on dist-test-slave-2dw5"
server_key: "730ed45ba79b6142b5e12a06adbfde7c"
server_key_iv: "c12f68f271be00b81cb8a71dbce291f9"
server_key_version: "encryptionkey@0"
I20260926 13:09:21.117551 6172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:21.138283 6172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:21.140148 6172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:21.140290 6172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:21.140515 6172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:21.140817 6172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:21.140851 6172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:21.140878 6172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:21.140903 6172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:21.146435 6172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37683
I20260926 13:09:21.146482 6300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37683 every 8 connection(s)
I20260926 13:09:21.146781 6172 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:21.153797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 6172
I20260926 13:09:21.153885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:21.154052 392 external_mini_cluster.cc:1428] Setting key 5924fe718db14b689fcb002c8795f456
I20260926 13:09:21.156002 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:21.147488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35573 (local address 127.0.98.62:33453)
0926 13:09:21.147532 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:21.147534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:21.147791 (+ 257us) server_negotiation.cc:407] Connection header received
0926 13:09:21.148744 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:21.148745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:21.148759 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:21.148810 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:21.149413 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.149910 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.150890 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.151099 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.151154 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:21.153180 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:21.153181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:21.153182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:21.153188 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:21.154627 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:21.155063 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:21.155064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:21.155065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:21.155117 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:21.155330 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:21.155331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:21.155332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:21.155436 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:21.155491 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:21.155761 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 13:09:21.155915 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:09:21.156113 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:21.147524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33453 (local address 127.0.98.2:35573)
0926 13:09:21.147670 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:21.147708 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:21.148634 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:21.148880 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:21.148883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:21.149135 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:21.149322 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:21.149326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.149994 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.149995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:21.150791 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:21.150793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.151168 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.151168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:21.151279 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:21.151722 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:21.151731 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:21.153095 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:21.154703 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:21.154711 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:21.154713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:21.154973 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:21.155172 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:21.155173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:21.155173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:21.155279 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:21.155487 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:21.155492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:21.155635 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 13:09:21.155728 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 13:09:21.155891 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260926 13:09:21.156517 6301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33453
I20260926 13:09:21.156653 6301 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:21.156821 6301 heartbeater.cc:511] Master 127.0.98.62:33453 requested a full tablet report, sending...
I20260926 13:09:21.157279 5976 ts_manager.cc:194] Registered new tserver with Master: 9908c3cf172a47a484c792d02e268217 (127.0.98.2:37683)
I20260926 13:09:21.157744 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35573
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:09:21.178045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33453
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:21.258530 6306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:21.258773 6306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:21.258805 6306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:21.260070 6306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:21.260129 6306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:21.260169 6306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:21.261736 6306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:21.261991 6306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:21.262200 6306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:09:21.267314 6306 server_base.cc:1034] running on GCE node
W20260926 13:09:21.267225 6312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:21.267248 6311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:21.267231 6314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:21.267726 6306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:21.267953 6306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:21.269088 6306 hybrid_clock.cc:648] HybridClock initialized: now 1790428161269072 us; error 30 us; skew 500 ppm
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:21.271582 6306 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:09:21.272063 6306 webserver.cc:460] Webserver started at http://127.0.98.3:42107/ using document root <none> and password file <none>
I20260926 13:09:21.272257 6306 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:21.272296 6306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:21.272392 6306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:21.273474 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "e55e4b886a2f4fcfb9261ee24b8c24f6"
format_stamp: "Formatted at 2026-09-26 13:09:21 on dist-test-slave-2dw5"
server_key: "2d242066cb2982c9846260bbc91825da"
server_key_iv: "3f64a93412121c8383944c1cb03c382f"
server_key_version: "encryptionkey@0"
I20260926 13:09:21.273799 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e55e4b886a2f4fcfb9261ee24b8c24f6"
format_stamp: "Formatted at 2026-09-26 13:09:21 on dist-test-slave-2dw5"
server_key: "2d242066cb2982c9846260bbc91825da"
server_key_iv: "3f64a93412121c8383944c1cb03c382f"
server_key_version: "encryptionkey@0"
I20260926 13:09:21.275125 6306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:21.275825 6321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:21.275974 6306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:21.276073 6306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "e55e4b886a2f4fcfb9261ee24b8c24f6"
format_stamp: "Formatted at 2026-09-26 13:09:21 on dist-test-slave-2dw5"
server_key: "2d242066cb2982c9846260bbc91825da"
server_key_iv: "3f64a93412121c8383944c1cb03c382f"
server_key_version: "encryptionkey@0"
I20260926 13:09:21.276136 6306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:21.293218 6306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:21.295262 6306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:21.295372 6306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:21.295558 6306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:21.295851 6306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:21.295885 6306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:21.295908 6306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:21.295933 6306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:21.301381 6306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37511
I20260926 13:09:21.301431 6434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37511 every 8 connection(s)
I20260926 13:09:21.301730 6306 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:21.302589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 6306
I20260926 13:09:21.302693 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:21.302881 392 external_mini_cluster.cc:1428] Setting key 070e0a4ce103a8e3ae484a91e3320ff0
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:21.310763 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:21.302463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54029 (local address 127.0.98.62:33453)
0926 13:09:21.302528 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:21.302530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:21.302728 (+ 198us) server_negotiation.cc:407] Connection header received
0926 13:09:21.303892 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:21.303893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:21.303904 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:21.303949 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:21.304596 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.305081 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.306042 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.306232 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.306285 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:21.308374 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:21.308375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:21.308375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:21.308383 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:21.309481 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:21.309836 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:21.309837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:21.309838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:21.309890 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:21.310099 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:21.310100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:21.310100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:21.310211 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:21.310275 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:21.310530 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 13:09:21.310666 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:09:21.310979 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:21.302422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33453 (local address 127.0.98.3:54029)
0926 13:09:21.302599 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:21.302643 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:21.303799 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:21.304024 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:21.304028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:21.304316 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:21.304503 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:21.304507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.305159 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.305160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:21.305959 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:21.305961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.306298 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.306299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:21.306427 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:21.306941 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:21.306949 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:21.308284 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:21.309555 (+ 1271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:21.309561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:21.309564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:21.309771 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:21.309943 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:21.309944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:21.309944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:21.310045 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:21.310264 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:21.310270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:21.310411 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 13:09:21.310502 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 13:09:21.310667 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260926 13:09:21.311390 6435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33453
I20260926 13:09:21.311481 6435 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:21.311627 6435 heartbeater.cc:511] Master 127.0.98.62:33453 requested a full tablet report, sending...
I20260926 13:09:21.311954 5976 ts_manager.cc:194] Registered new tserver with Master: e55e4b886a2f4fcfb9261ee24b8c24f6 (127.0.98.3:37511)
I20260926 13:09:21.312376 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54029
I20260926 13:09:21.314584 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/test-admin.keytab.
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:21.331699 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 13:09:21.333134 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:21.332485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.0.98.1:37917)
0926 13:09:21.332636 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:21.332642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:21.332653 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:21.332678 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:21.332680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:21.332693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:21.332759 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:21.333057 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59056: BlockingRecv error: recv got EOF from 127.0.0.1:59056 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
W20260926 13:09:21.333266 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:21.334332 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:21.335192 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:09:21.344830 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:21.335708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.0.98.1:37917)
0926 13:09:21.335980 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:21.335982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:21.335989 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:09:21.336012 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:21.336013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:21.336025 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:21.336072 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:21.336539 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.337685 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.338707 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.339218 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.339276 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:21.341739 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:21.341749 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:21.341749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:21.341756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:21.343577 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:21.343958 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:21.343962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:21.343962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:21.344021 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:21.344226 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:21.344226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:21.344227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:21.344403 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:21.344540 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:21.344633 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 13:09:21.344721 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250}
W20260926 13:09:21.346906 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:21.346211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.0.98.1:37917)
0926 13:09:21.346247 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:21.346249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:21.346387 (+ 138us) server_negotiation.cc:407] Connection header received
0926 13:09:21.346454 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:21.346455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:21.346468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:21.346530 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:21.346863 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59078: BlockingRecv error: recv got EOF from 127.0.0.1:59078 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 13:09:21.346961 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:21.348060 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:21.348912 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:09:21.355147 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:21.349388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.0.98.1:37917)
0926 13:09:21.349428 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:21.349429 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:21.349438 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:21.349488 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:21.349489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:21.349495 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:21.349531 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:21.349813 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.350290 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.351066 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.351259 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.351307 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:21.353069 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:21.353070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:21.353070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:21.353077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:21.354272 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:21.354535 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:21.354536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:21.354536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:21.354646 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:21.354773 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:21.354773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:21.354774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:21.354879 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:21.354957 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:21.355029 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 13:09:21.355088 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 13:09:21.356564 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:21.356030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.0.98.1:37917)
0926 13:09:21.356065 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:21.356067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:21.356134 (+ 67us) server_negotiation.cc:407] Connection header received
0926 13:09:21.356190 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:21.356190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:21.356198 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:21.356244 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:21.356522 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59106: BlockingRecv error: recv got EOF from 127.0.0.1:59106 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 13:09:21.356726 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428136114168-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:21.357671 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:21.358503 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:09:21.364878 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:21.358983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59112 (local address 127.0.98.1:37917)
0926 13:09:21.359035 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:21.359036 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:21.359083 (+ 47us) server_negotiation.cc:407] Connection header received
0926 13:09:21.359132 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:21.359133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:21.359140 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:21.359185 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:21.359556 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.360033 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.360719 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:21.360900 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:21.360947 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:21.362801 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:21.362802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:21.362802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:21.362809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:21.364007 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:21.364309 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:21.364310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:21.364310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:21.364352 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:21.364486 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:21.364487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:21.364487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:21.364569 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:21.364650 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:21.364741 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 13:09:21.364794 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:09:21.365474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 6038
I20260926 13:09:21.369843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 6172
I20260926 13:09:21.373457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 6306
I20260926 13:09:21.376868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 5944
2026-09-26T13:09:21Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1796 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[6455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[6455](info): set up 2 sockets
Sep 26 13:09:21 dist-test-slave-2dw5 krb5kdc[6455](info): commencing operation
krb5kdc: starting...
W20260926 13:09:22.327051 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:22 dist-test-slave-2dw5 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 13:09:22.442119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45001
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:22.521955 6468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:22.522178 6468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:22.522208 6468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:22.523411 6468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:22.523459 6468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:22.523473 6468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:22.523484 6468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:22.523495 6468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:22.525024 6468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:22.525247 6468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:22.525432 6468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:22.530571 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:22.530548 6474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:22.530542 6473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:22.530609 6468 server_base.cc:1034] running on GCE node
I20260926 13:09:22.531256 6468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:22.531534 6468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:22.532711 6468 hybrid_clock.cc:648] HybridClock initialized: now 1790428162532686 us; error 35 us; skew 500 ppm
Sep 26 13:09:22 dist-test-slave-2dw5 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428162, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:22.535274 6468 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 13:09:22.535706 6468 webserver.cc:460] Webserver started at http://127.0.98.62:44597/ using document root <none> and password file <none>
I20260926 13:09:22.535943 6468 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:22.536008 6468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:22.536151 6468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:22.537274 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "f40517c7135f4dd6b4f86d931669cc3f"
format_stamp: "Formatted at 2026-09-26 13:09:22 on dist-test-slave-2dw5"
server_key: "bed8c532fc19bd09b33250c67bebbd3c"
server_key_iv: "5870fa7f4009dd46d06ad1488d2ac2da"
server_key_version: "encryptionkey@0"
I20260926 13:09:22.537608 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "f40517c7135f4dd6b4f86d931669cc3f"
format_stamp: "Formatted at 2026-09-26 13:09:22 on dist-test-slave-2dw5"
server_key: "bed8c532fc19bd09b33250c67bebbd3c"
server_key_iv: "5870fa7f4009dd46d06ad1488d2ac2da"
server_key_version: "encryptionkey@0"
I20260926 13:09:22.538938 6468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:22.539673 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:22.539899 6468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:22.539988 6468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "f40517c7135f4dd6b4f86d931669cc3f"
format_stamp: "Formatted at 2026-09-26 13:09:22 on dist-test-slave-2dw5"
server_key: "bed8c532fc19bd09b33250c67bebbd3c"
server_key_iv: "5870fa7f4009dd46d06ad1488d2ac2da"
server_key_version: "encryptionkey@0"
I20260926 13:09:22.540043 6468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:22.572810 6468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:22.574712 6468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:22.574833 6468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:22.578588 6468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45001
I20260926 13:09:22.578626 6535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45001 every 8 connection(s)
I20260926 13:09:22.579075 6468 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:22.580130 6536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:22.582377 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f: Bootstrap starting.
I20260926 13:09:22.582973 6536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:22.583237 6536 log.cc:824] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:22.583904 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f: No bootstrap required, opened a new log
I20260926 13:09:22.585549 6536 raft_consensus.cc:348] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } }
I20260926 13:09:22.585742 6536 raft_consensus.cc:374] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:22.585768 6536 raft_consensus.cc:729] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40517c7135f4dd6b4f86d931669cc3f, State: Initialized, Role: FOLLOWER
I20260926 13:09:22.585906 6536 consensus_queue.cc:260] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } }
I20260926 13:09:22.585978 6536 raft_consensus.cc:388] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:22.586002 6536 raft_consensus.cc:482] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:22.586048 6536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:22.587101 6536 raft_consensus.cc:504] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } }
I20260926 13:09:22.587249 6536 leader_election.cc:302] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f40517c7135f4dd6b4f86d931669cc3f; no voters:
I20260926 13:09:22.587304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 6468
I20260926 13:09:22.587410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:22.587435 6536 leader_election.cc:288] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:22.587464 6539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:22.587622 6539 raft_consensus.cc:686] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 1 LEADER]: Becoming Leader. State: Replica: f40517c7135f4dd6b4f86d931669cc3f, State: Running, Role: LEADER
I20260926 13:09:22.587661 392 external_mini_cluster.cc:1428] Setting key 94f2ef18d633972399187aec51c19716
I20260926 13:09:22.587713 6536 sys_catalog.cc:564] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:22.587817 6539 consensus_queue.cc:237] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } }
I20260926 13:09:22.588215 6540 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } }
I20260926 13:09:22.588380 6540 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:22.588572 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f40517c7135f4dd6b4f86d931669cc3f. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } }
I20260926 13:09:22.588572 6550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:22.588671 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:22.589185 6550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:22.591946 6550 catalog_manager.cc:1347] Generated new cluster ID: 412444a8b1f04915b5763a001a60229f
I20260926 13:09:22.592013 6550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:09:22 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:22.598359 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:22.588289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.0.98.62:45001)
0926 13:09:22.588501 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:22.588512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:22.589182 (+ 670us) server_negotiation.cc:407] Connection header received
0926 13:09:22.589514 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:22.589522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:22.589579 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:22.589670 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:22.590303 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:22.591724 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:22.592607 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:22.593057 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:22.593176 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:22.595331 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:22.595336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:22.595336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:22.595349 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:22.596923 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:22.597240 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:22.597243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:22.597243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:22.597341 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:22.597507 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:22.597508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:22.597508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:22.597758 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:22.597898 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:22.597993 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 13:09:22.598143 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260926 13:09:22.607869 6550 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:22.608616 6550 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:22.614797 6550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f: Generated new TSK 0
I20260926 13:09:22.615025 6550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 13:09:22.641880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45001
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:22.746970 6562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:22.747166 6562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:22.747190 6562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:22.748369 6562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:22.748414 6562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:22.748441 6562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:22.749966 6562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:22.750193 6562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:22.750411 6562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:22.755405 6570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:22.755414 6568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:22.755415 6567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:22.755472 6562 server_base.cc:1034] running on GCE node
I20260926 13:09:22.755810 6562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:22.756042 6562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:22.757184 6562 hybrid_clock.cc:648] HybridClock initialized: now 1790428162757172 us; error 28 us; skew 500 ppm
Sep 26 13:09:22 dist-test-slave-2dw5 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428162, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:22.759545 6562 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 13:09:22.760030 6562 webserver.cc:460] Webserver started at http://127.0.98.1:45219/ using document root <none> and password file <none>
I20260926 13:09:22.760250 6562 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:22.760298 6562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:22.760421 6562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:22.761445 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "2caca43a229c408aa688a866f07250b6"
format_stamp: "Formatted at 2026-09-26 13:09:22 on dist-test-slave-2dw5"
server_key: "947a430f3704185150cb5ca9cbefabc9"
server_key_iv: "5dedb8925aea45f6a5992c72abfdc699"
server_key_version: "encryptionkey@0"
I20260926 13:09:22.761770 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2caca43a229c408aa688a866f07250b6"
format_stamp: "Formatted at 2026-09-26 13:09:22 on dist-test-slave-2dw5"
server_key: "947a430f3704185150cb5ca9cbefabc9"
server_key_iv: "5dedb8925aea45f6a5992c72abfdc699"
server_key_version: "encryptionkey@0"
I20260926 13:09:22.763049 6562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:09:22.763772 6577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:22.763936 6562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:22.764015 6562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "2caca43a229c408aa688a866f07250b6"
format_stamp: "Formatted at 2026-09-26 13:09:22 on dist-test-slave-2dw5"
server_key: "947a430f3704185150cb5ca9cbefabc9"
server_key_iv: "5dedb8925aea45f6a5992c72abfdc699"
server_key_version: "encryptionkey@0"
I20260926 13:09:22.764081 6562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:22.782124 6562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:22.784133 6562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:22.784282 6562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:22.784502 6562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:22.784790 6562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:22.784823 6562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:22.784852 6562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:22.784874 6562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:22.790637 6562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40611
I20260926 13:09:22.790678 6690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40611 every 8 connection(s)
I20260926 13:09:22.791000 6562 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:09:22 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428162, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:22.797637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 6562
I20260926 13:09:22.797730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:22.797902 392 external_mini_cluster.cc:1428] Setting key be5069251d2e327b7ae17683e1c581e3
I20260926 13:09:22.799780 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:22.791744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51439 (local address 127.0.98.62:45001)
0926 13:09:22.791829 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:22.791831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:22.791981 (+ 150us) server_negotiation.cc:407] Connection header received
0926 13:09:22.792451 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:22.792452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:22.792464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:22.792527 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:22.793392 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:22.793889 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:22.794934 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:22.795133 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:22.795187 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:22.797128 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:22.797130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:22.797130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:22.797137 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:22.798524 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:22.798879 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:22.798880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:22.798880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:22.798937 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:22.799150 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:22.799151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:22.799151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:22.799257 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:22.799310 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:22.799586 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 13:09:22.799676 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:09:22.799940 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:22.791698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.1:51439)
0926 13:09:22.791858 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:22.791900 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:22.792373 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:22.792589 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:22.792592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:22.793131 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:22.793321 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:22.793325 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:22.793972 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:22.793973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:22.794808 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:22.794810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:22.795211 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:22.795212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:22.795319 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:22.795728 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:22.795736 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:22.797031 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:22.798593 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:22.798599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:22.798601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:22.798799 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:22.798988 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:22.798989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:22.798989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:22.799096 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:22.799307 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:22.799313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:22.799452 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 13:09:22.799554 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 13:09:22.799723 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260926 13:09:22.800359 6691 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45001
I20260926 13:09:22.800448 6691 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:22.800611 6691 heartbeater.cc:511] Master 127.0.98.62:45001 requested a full tablet report, sending...
I20260926 13:09:22.801088 6500 ts_manager.cc:194] Registered new tserver with Master: 2caca43a229c408aa688a866f07250b6 (127.0.98.1:40611)
I20260926 13:09:22.801678 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51439
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 13:09:22.823484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45001
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:22.903846 6696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:22.904062 6696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:22.904086 6696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:22.905285 6696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:22.905332 6696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:22.905359 6696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:22.906896 6696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:22.907119 6696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:22.907346 6696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:22.912324 6704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:22.912331 6702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:22.912331 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:22.912600 6696 server_base.cc:1034] running on GCE node
I20260926 13:09:22.912832 6696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:22.913069 6696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:22.914211 6696 hybrid_clock.cc:648] HybridClock initialized: now 1790428162914190 us; error 37 us; skew 500 ppm
Sep 26 13:09:22 dist-test-slave-2dw5 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428162, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:22.916755 6696 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 13:09:22.917169 6696 webserver.cc:460] Webserver started at http://127.0.98.2:37413/ using document root <none> and password file <none>
I20260926 13:09:22.917343 6696 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:22.917382 6696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:22.917486 6696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:22.918538 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "7bb70baa07584f89868a7f334975c060"
format_stamp: "Formatted at 2026-09-26 13:09:22 on dist-test-slave-2dw5"
server_key: "974876946fa12b2ce4eb3cf76ea48454"
server_key_iv: "3648be212b69953bc28776c1225cf547"
server_key_version: "encryptionkey@0"
I20260926 13:09:22.918872 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7bb70baa07584f89868a7f334975c060"
format_stamp: "Formatted at 2026-09-26 13:09:22 on dist-test-slave-2dw5"
server_key: "974876946fa12b2ce4eb3cf76ea48454"
server_key_iv: "3648be212b69953bc28776c1225cf547"
server_key_version: "encryptionkey@0"
I20260926 13:09:22.920099 6696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:22.920792 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:22.920929 6696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:09:22.921005 6696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "7bb70baa07584f89868a7f334975c060"
format_stamp: "Formatted at 2026-09-26 13:09:22 on dist-test-slave-2dw5"
server_key: "974876946fa12b2ce4eb3cf76ea48454"
server_key_iv: "3648be212b69953bc28776c1225cf547"
server_key_version: "encryptionkey@0"
I20260926 13:09:22.921059 6696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:22.931954 6696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:22.933908 6696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:22.934046 6696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:22.934280 6696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:22.934535 6696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:22.934563 6696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:22.934584 6696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:22.934600 6696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:22.940225 6696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44653
I20260926 13:09:22.940274 6824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44653 every 8 connection(s)
I20260926 13:09:22.940593 6696 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:09:22 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428162, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:22.947974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 6696
I20260926 13:09:22.948098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:22.948302 392 external_mini_cluster.cc:1428] Setting key bd625cbe458b0106cec116dd448eae7e
I20260926 13:09:22.949324 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:22.941258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51631 (local address 127.0.98.62:45001)
0926 13:09:22.941304 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:22.941306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:22.941484 (+ 178us) server_negotiation.cc:407] Connection header received
0926 13:09:22.941971 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:22.941972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:22.941982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:22.942023 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:22.943079 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:22.943596 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:22.944626 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:22.944825 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:22.944877 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:22.946780 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:22.946781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:22.946781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:22.946788 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:22.948047 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:22.948461 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:22.948462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:22.948463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:22.948516 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:22.948737 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:22.948738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:22.948745 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:22.948852 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:22.948905 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:22.949156 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 13:09:22.949236 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:09:22.949512 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:22.941225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.2:51631)
0926 13:09:22.941372 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:22.941406 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:22.941894 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:22.942094 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:22.942098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:22.942705 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:22.942992 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:22.942997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:22.943670 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:22.943670 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:22.944542 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:22.944545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:22.944889 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:22.944890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:22.944988 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:22.945419 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:22.945427 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:22.946705 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:22.948124 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:22.948132 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:22.948134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:22.948399 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:22.948568 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:22.948568 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:22.948569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:22.948683 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:22.948904 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:22.948911 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:22.949044 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 13:09:22.949130 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 13:09:22.949296 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260926 13:09:22.949899 6825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45001
I20260926 13:09:22.949985 6825 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:22.950152 6825 heartbeater.cc:511] Master 127.0.98.62:45001 requested a full tablet report, sending...
I20260926 13:09:22.950505 6500 ts_manager.cc:194] Registered new tserver with Master: 7bb70baa07584f89868a7f334975c060 (127.0.98.2:44653)
I20260926 13:09:22.950889 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51631
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 13:09:22.973237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45001
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:23.053361 6830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:23.053582 6830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:23.053606 6830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:23.054785 6830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:23.054833 6830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:23.054910 6830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:23.056474 6830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:23.056699 6830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:23.056908 6830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:23.061961 6836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:23.062034 6830 server_base.cc:1034] running on GCE node
W20260926 13:09:23.061909 6838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:23.061919 6835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:23.062321 6830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:23.062551 6830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:23.063704 6830 hybrid_clock.cc:648] HybridClock initialized: now 1790428163063688 us; error 27 us; skew 500 ppm
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:23.066334 6830 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 13:09:23.066890 6830 webserver.cc:460] Webserver started at http://127.0.98.3:39047/ using document root <none> and password file <none>
I20260926 13:09:23.067104 6830 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:23.067150 6830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:23.067276 6830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:23.068405 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "c8137509b54b41d78e847082f20b7fb5"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "955cb7a087ee056ba174cd6fcbdb23ce"
server_key_iv: "88ef78a5272a2d24544bb0d3d9313cc3"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.068748 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c8137509b54b41d78e847082f20b7fb5"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "955cb7a087ee056ba174cd6fcbdb23ce"
server_key_iv: "88ef78a5272a2d24544bb0d3d9313cc3"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.070004 6830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:09:23.070873 6845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.071038 6830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:23.071125 6830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "c8137509b54b41d78e847082f20b7fb5"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "955cb7a087ee056ba174cd6fcbdb23ce"
server_key_iv: "88ef78a5272a2d24544bb0d3d9313cc3"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.071193 6830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:23.082584 6830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:23.084471 6830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:23.084604 6830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:23.084805 6830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:23.085085 6830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:23.085119 6830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.085147 6830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:23.085171 6830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.090978 6830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35143
I20260926 13:09:23.091017 6958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35143 every 8 connection(s)
I20260926 13:09:23.091328 6830 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:23.097692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 6830
I20260926 13:09:23.097786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:23.097945 392 external_mini_cluster.cc:1428] Setting key bf769d8aadc42f418b5ee745e1f109e4
I20260926 13:09:23.100085 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.092021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57891 (local address 127.0.98.62:45001)
0926 13:09:23.092077 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.092079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.092264 (+ 185us) server_negotiation.cc:407] Connection header received
0926 13:09:23.092792 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.092793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.092806 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.092857 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.093798 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.094280 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.095236 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.095419 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.095473 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.097388 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.097390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.097390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.097397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.098911 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.099250 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.099252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.099252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.099306 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.099511 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.099512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.099513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.099621 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.099673 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.099924 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.099997 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:09:23.100253 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.092009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.3:57891)
0926 13:09:23.092151 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.092186 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.092669 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.092936 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.092941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.093508 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.093696 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.093701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.094359 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.094360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.095157 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.095158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.095484 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.095485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.095596 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.096045 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.096053 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.097305 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.098975 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.098983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.098985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.099190 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.099357 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.099358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.099358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.099460 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.099671 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.099677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.099811 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.099897 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.100047 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260926 13:09:23.100643 6959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45001
I20260926 13:09:23.100737 6959 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:23.100906 6959 heartbeater.cc:511] Master 127.0.98.62:45001 requested a full tablet report, sending...
I20260926 13:09:23.101241 6500 ts_manager.cc:194] Registered new tserver with Master: c8137509b54b41d78e847082f20b7fb5 (127.0.98.3:35143)
I20260926 13:09:23.101601 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57891
I20260926 13:09:23.109655 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 13:09:23.137310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45555
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45001,127.0.98.61:45555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:23.218539 6964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:23.218768 6964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:23.218791 6964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:23.220011 6964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:23.220063 6964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:23.220077 6964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:23.220108 6964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:23.220131 6964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:23.221669 6964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45001,127.0.98.61:45555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
W20260926 13:09:23.221781 6964 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45001,127.0.98.61:45555'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 13:09:23.221928 6964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:23.222157 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:23.227257 6969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:23.227210 6970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:23.227211 6972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:23.227465 6964 server_base.cc:1034] running on GCE node
I20260926 13:09:23.227700 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:23.228014 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:23.229174 6964 hybrid_clock.cc:648] HybridClock initialized: now 1790428163229154 us; error 33 us; skew 500 ppm
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:23.231905 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 13:09:23.232411 6964 webserver.cc:460] Webserver started at http://127.0.98.61:42431/ using document root <none> and password file <none>
I20260926 13:09:23.232645 6964 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:23.232693 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:23.232831 6964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:23.233896 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data/instance:
uuid: "c54404f73c1e40229892ba9529e9a003"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "62988eb1e1354ffcdf552aa7abf26a82"
server_key_iv: "2fcd3d6742cf3df09429ab993f365206"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.234218 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/wal/instance:
uuid: "c54404f73c1e40229892ba9529e9a003"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "62988eb1e1354ffcdf552aa7abf26a82"
server_key_iv: "2fcd3d6742cf3df09429ab993f365206"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.235487 6964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 13:09:23.236131 6979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.236281 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:23.236359 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/wal
uuid: "c54404f73c1e40229892ba9529e9a003"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "62988eb1e1354ffcdf552aa7abf26a82"
server_key_iv: "2fcd3d6742cf3df09429ab993f365206"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.236419 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:23.263932 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:23.266034 6964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:23.266170 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:23.270056 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45555
I20260926 13:09:23.270094 7031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45555 every 8 connection(s)
I20260926 13:09:23.270430 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
I20260926 13:09:23.271526 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:23.272271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 6964
I20260926 13:09:23.273376 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 13:09:23.282115 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.274124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.0.98.62:45001)
0926 13:09:23.274182 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.274184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.274390 (+ 206us) server_negotiation.cc:407] Connection header received
0926 13:09:23.274457 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.274459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.274470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.274525 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.274985 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.275784 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.277001 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.277277 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.277332 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.279552 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.279553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.279554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.279560 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.280858 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.281221 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.281222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.281222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.281273 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.281485 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.281486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.281487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.281633 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.281691 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.281945 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.282024 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 13:09:23.282281 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.274049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33046)
0926 13:09:23.274254 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.274296 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.274394 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.274537 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.274539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.274775 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.274909 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.274914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.275880 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.275881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.276927 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.276930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.277369 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.277370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.277435 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.277883 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.277893 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.279468 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.280932 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.280938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.280940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.281158 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.281331 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.281332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.281332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.281434 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.281690 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.281696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.281833 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.281918 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.282069 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
I20260926 13:09:23.282807 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.272884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.0.98.61:45555)
0926 13:09:23.273038 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.273046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.273071 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:09:23.273497 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.273504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.273551 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.273659 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.274431 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.275774 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.276831 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.277211 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.277324 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.279994 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.280000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.280001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.280007 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.281458 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.281808 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.281811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.281811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.281859 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.282023 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.282024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.282025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.282149 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.282291 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.282670 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.282742 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260926 13:09:23.282873 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 13:09:23.288429 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.283115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45555 (local address 127.0.0.1:45692)
0926 13:09:23.283154 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.283163 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.283202 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.283315 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.283315 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.283466 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.283510 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.283511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.284086 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.284087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.284565 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.284567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.284896 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.284896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.284953 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.285298 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.285302 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.286368 (+ 1066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.287577 (+ 1209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.287578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.287578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.287717 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.287911 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.287911 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.287911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.287964 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.288162 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.288163 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.288214 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.288276 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.288364 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 13:09:23.288523 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.283161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.0.98.61:45555)
0926 13:09:23.283193 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.283194 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.283202 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:23.283244 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.283245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.283251 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.283299 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.283572 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.284027 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.284628 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.284842 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.284894 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.286434 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.286435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.286435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.286441 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.287509 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.287768 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.287769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.287769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.287841 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.288011 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.288012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.288012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.288113 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.288164 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.288315 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.288423 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":16}
I20260926 13:09:23.289885 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: Bootstrap starting.
I20260926 13:09:23.290625 7032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:23.290880 7032 log.cc:824] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:23.291560 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: No bootstrap required, opened a new log
I20260926 13:09:23.292928 7032 raft_consensus.cc:348] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } }
I20260926 13:09:23.293061 7032 raft_consensus.cc:374] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:23.293081 7032 raft_consensus.cc:729] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c54404f73c1e40229892ba9529e9a003, State: Initialized, Role: FOLLOWER
I20260926 13:09:23.293179 7032 consensus_queue.cc:260] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } }
I20260926 13:09:23.293426 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } } }
I20260926 13:09:23.293483 7032 sys_catalog.cc:564] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:23.293519 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 13:09:23.295176 7049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:09:23.295233 7049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:09:23.296438 7038 raft_consensus.cc:482] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:23.296514 7038 raft_consensus.cc:504] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } }
I20260926 13:09:23.296715 7038 leader_election.cc:288] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f40517c7135f4dd6b4f86d931669cc3f (127.0.98.62:45001)
I20260926 13:09:23.296897 6964 master_runner.cc:428] Detected that this master c54404f73c1e40229892ba9529e9a003 is joining an existing cluster
I20260926 13:09:23.296947 6964 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45555
I20260926 13:09:23.296937 6510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c54404f73c1e40229892ba9529e9a003" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40517c7135f4dd6b4f86d931669cc3f" is_pre_election: true
I20260926 13:09:23.297277 6980 leader_election.cc:302] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c54404f73c1e40229892ba9529e9a003; no voters: f40517c7135f4dd6b4f86d931669cc3f
I20260926 13:09:23.297420 7038 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 13:09:23.304296 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.298047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.0.98.61:45555)
0926 13:09:23.298082 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.298084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.298146 (+ 62us) server_negotiation.cc:407] Connection header received
0926 13:09:23.298199 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.298200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.298208 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.298252 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.298732 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.299482 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.300545 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.300772 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.300824 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.301890 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.301891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.301892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.301897 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.303330 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.303621 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.303622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.303622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.303676 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.303823 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.303824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.303824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.303927 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.303980 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.304097 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.304173 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:09:23.304440 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.297977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33056 (local address 127.0.98.62:45001)
0926 13:09:23.298021 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.298024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.298154 (+ 130us) server_negotiation.cc:407] Connection header received
0926 13:09:23.298265 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.298266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.298277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.298330 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.298773 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.299514 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.300358 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.300561 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.300621 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.301600 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.301601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.301602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.301609 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.303097 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.303504 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.303505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.303506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.303561 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.303880 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.303881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.303881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.304017 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.304061 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.304259 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.304351 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:09:23.304641 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.297907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45555 (local address 127.0.0.1:45708)
0926 13:09:23.298100 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.298109 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.298159 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.298373 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.298374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.298603 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.298669 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.298671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.299557 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.299558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.300469 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.300471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.300833 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.300834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.301121 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.301452 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.301456 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.301819 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.303387 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.303388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.303388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.303557 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.303725 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.303725 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.303725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.303767 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.303978 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.303979 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.304014 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.304073 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.304222 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":92,"thread_start_us":89,"threads_started":1}
I20260926 13:09:23.304760 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.297902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33056)
0926 13:09:23.298087 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.298100 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.298177 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.298350 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.298351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.298643 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.298711 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.298713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.299589 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.299589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.300191 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.300193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.300628 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.300629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.300688 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.301103 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.301108 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.301513 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.303191 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.303192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.303193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.303434 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.303746 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.303746 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.303746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.303797 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.304106 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.304107 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.304156 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.304528 (+ 372us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.304638 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":19,"thread_start_us":93,"threads_started":1}
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428162, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 13:09:23.313360 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.306907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.0.98.61:45555)
0926 13:09:23.306940 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.306942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.307083 (+ 141us) server_negotiation.cc:407] Connection header received
0926 13:09:23.307161 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.307162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.307172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.307215 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.307617 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.308076 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.308791 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.308977 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.309029 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.311151 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.311152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.311152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.311159 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.312345 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.312667 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.312668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.312668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.312716 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.312878 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.312879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.312879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.312976 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.313027 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.313181 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.313263 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 13:09:23.313375 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.306833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45555 (local address 127.0.0.1:45724)
0926 13:09:23.306991 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.307029 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.307096 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.307220 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.307224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.307499 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.307547 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.307549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.308147 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.308149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.308715 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.308716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.309040 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.309041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.309126 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.309513 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.309518 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.311067 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.312417 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.312424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.312426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.312606 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.312767 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.312768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.312769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.312832 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.313027 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.313034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.313085 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.313155 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.313267 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20260926 13:09:23.314144 6500 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45555: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c54404f73c1e40229892ba9529e9a003"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45555
}
attrs {
promote: true
}
}
dest_uuid: "f40517c7135f4dd6b4f86d931669cc3f"
cas_config_opid_index: -1
I20260926 13:09:23.314433 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true } }
I20260926 13:09:23.315584 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c54404f73c1e40229892ba9529e9a003 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true } } }
I20260926 13:09:23.315857 7006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:23.316606 7006 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 FOLLOWER]: Refusing update from remote peer f40517c7135f4dd6b4f86d931669cc3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 13:09:23.316663 7059 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45555
I20260926 13:09:23.316747 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true } } }
I20260926 13:09:23.316810 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:23.316978 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:23.317118 6964 master.cc:537] Master@127.0.98.61:45555 shutting down...
W20260926 13:09:23.318797 6485 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 13:09:23.319677 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f -> Peer c54404f73c1e40229892ba9529e9a003 (127.0.98.61:45555): Couldn't send request to peer c54404f73c1e40229892ba9529e9a003. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 13:09:23.320230 6964 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:23.320314 6964 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:23.320374 6964 tablet_replica.cc:330] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: stopping tablet replica
I20260926 13:09:23.333195 6964 master.cc:552] Master@127.0.98.61:45555 shutdown complete.
I20260926 13:09:23.333253 6964 master_runner.cc:305] Clearing existing system tablet
I20260926 13:09:23.333375 6964 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:23.333415 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:23.334023 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:23.334475 7063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.334606 6964 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:23.334661 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.334684 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/wal
uuid: "c54404f73c1e40229892ba9529e9a003"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "62988eb1e1354ffcdf552aa7abf26a82"
server_key_iv: "2fcd3d6742cf3df09429ab993f365206"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.335405 6964 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:23.337000 6964 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 13:09:23.337073 6964 log.cc:1196] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 13:09:23.337261 6964 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: Deleting consensus metadata
I20260926 13:09:23.337396 6964 master_runner.cc:315] Copying system tablet from 127.0.98.62:45001
I20260926 13:09:23.338023 6964 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45001
I20260926 13:09:23.345198 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.339288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.0.98.62:45001)
0926 13:09:23.339517 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.339520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.339530 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:23.339576 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.339577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.339588 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.339658 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.340041 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.340777 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.341796 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.342003 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.342058 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.342985 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.342987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.342987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.342994 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.344207 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.344516 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.344517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.344518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.344571 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.344740 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.344740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.344741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.344848 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.344900 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.345027 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.345101 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260926 13:09:23.345214 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.339233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33064)
0926 13:09:23.339355 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.339365 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.339417 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.339672 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.339673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.339872 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.339942 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.339945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.340887 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.340888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.341727 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.341730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.342061 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.342062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.342123 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.342580 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.342584 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.342909 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.344278 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.344279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.344279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.344453 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.344625 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.344626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.344626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.344691 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.344897 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.344898 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.344942 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.345000 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.345103 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260926 13:09:23.345551 6520 tablet_copy_service.cc:140] P f40517c7135f4dd6b4f86d931669cc3f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c54404f73c1e40229892ba9529e9a003 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33064)
I20260926 13:09:23.345642 6520 tablet_copy_service.cc:161] P f40517c7135f4dd6b4f86d931669cc3f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c54404f73c1e40229892ba9529e9a003 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33064: session id = c54404f73c1e40229892ba9529e9a003-00000000000000000000000000000000
I20260926 13:09:23.346582 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 13:09:23.347294 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 13:09:23.348549 6964 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: tablet copy: Starting download of 0 data blocks...
I20260926 13:09:23.348709 6964 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: tablet copy: Starting download of 1 WAL segments...
I20260926 13:09:23.349570 6964 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 13:09:23.350476 6520 tablet_copy_service.cc:342] P f40517c7135f4dd6b4f86d931669cc3f: Request end of tablet copy session c54404f73c1e40229892ba9529e9a003-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33064
I20260926 13:09:23.350535 6520 tablet_copy_service.cc:433] P f40517c7135f4dd6b4f86d931669cc3f: ending tablet copy session c54404f73c1e40229892ba9529e9a003-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c54404f73c1e40229892ba9529e9a003
I20260926 13:09:23.352814 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:23.354327 7071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:23.354429 7070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:23.354532 7073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:23.354619 6964 server_base.cc:1034] running on GCE node
I20260926 13:09:23.354696 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:23.354903 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:23.356024 6964 hybrid_clock.cc:648] HybridClock initialized: now 1790428163356019 us; error 26 us; skew 500 ppm
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:23.357354 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 13:09:23.357614 6964 webserver.cc:460] Webserver started at http://127.0.98.61:42321/ using document root <none> and password file <none>
I20260926 13:09:23.357724 6964 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:23.357758 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:23.358413 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:09:23.358932 7080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.359074 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.359123 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/wal
uuid: "c54404f73c1e40229892ba9529e9a003"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "62988eb1e1354ffcdf552aa7abf26a82"
server_key_iv: "2fcd3d6742cf3df09429ab993f365206"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.359179 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:23.375972 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:23.377590 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:23.380635 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45555
I20260926 13:09:23.380841 7132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45555 every 8 connection(s)
I20260926 13:09:23.381013 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
I20260926 13:09:23.381719 7133 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:09:23.382105 7133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: Bootstrap starting.
I20260926 13:09:23.384395 7133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:09:23.384621 7133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: Bootstrap complete.
I20260926 13:09:23.384861 7133 raft_consensus.cc:348] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true } }
I20260926 13:09:23.384954 7133 raft_consensus.cc:729] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c54404f73c1e40229892ba9529e9a003, State: Initialized, Role: LEARNER
I20260926 13:09:23.384994 7133 consensus_queue.cc:260] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true } }
I20260926 13:09:23.385170 7133 sys_catalog.cc:564] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:23.386786 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true } } }
I20260926 13:09:23.387004 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: This master's current role is: LEARNER
I20260926 13:09:23.387210 7148 catalog_manager.cc:1259] Loaded cluster ID: 412444a8b1f04915b5763a001a60229f
I20260926 13:09:23.387292 7148 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: loading cluster ID for follower catalog manager: success
I20260926 13:09:23.388460 7148 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: acquiring CA information for follower catalog manager: success
I20260926 13:09:23.388866 7148 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:09:23.391448 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.384359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.0.98.61:45555)
0926 13:09:23.384598 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.384600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.384611 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:23.384637 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.384638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.384649 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.385416 (+ 767us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.385451 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.386458 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.387432 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.387654 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.387709 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.388951 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.388952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.388952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.388960 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.390386 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.390700 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.390701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.390702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.390763 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.390958 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.390958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.390959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.391073 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.391168 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.391261 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.391354 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 13:09:23.417975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35631
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45001,127.0.98.61:45555,127.0.98.60:35631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:23.512048 7151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:23.512269 7151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:23.512291 7151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:23.513535 7151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:23.513586 7151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:23.513599 7151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:23.513612 7151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:23.513622 7151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:23.515540 7151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45001,127.0.98.61:45555,127.0.98.60:35631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:23.515822 7151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:23.516065 7151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:23.521066 7157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:23.521066 7159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:23.521075 7156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:23.521303 7151 server_base.cc:1034] running on GCE node
I20260926 13:09:23.521596 7151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:23.521881 7151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:23.523015 7151 hybrid_clock.cc:648] HybridClock initialized: now 1790428163523002 us; error 29 us; skew 500 ppm
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:23.533778 7151 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 13:09:23.534224 7151 webserver.cc:460] Webserver started at http://127.0.98.60:38131/ using document root <none> and password file <none>
I20260926 13:09:23.534421 7151 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:23.534467 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:23.534567 7151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:23.535688 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data/instance:
uuid: "dea95093e5a546bda5ade0b7aa86c648"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "1a588826284390f620f2bcd0a0d7ca27"
server_key_iv: "9d886d517ecd15bf181983f8427000d6"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.536001 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/wal/instance:
uuid: "dea95093e5a546bda5ade0b7aa86c648"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "1a588826284390f620f2bcd0a0d7ca27"
server_key_iv: "9d886d517ecd15bf181983f8427000d6"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.537248 7151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:23.537988 7166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.538241 7151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:23.538329 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/wal
uuid: "dea95093e5a546bda5ade0b7aa86c648"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "1a588826284390f620f2bcd0a0d7ca27"
server_key_iv: "9d886d517ecd15bf181983f8427000d6"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.538409 7151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:23.545450 7151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:23.547286 7151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:23.547394 7151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:23.551914 7151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35631
I20260926 13:09:23.551955 7218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35631 every 8 connection(s)
I20260926 13:09:23.552366 7151 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
I20260926 13:09:23.553303 7219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:23.553360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 7151
I20260926 13:09:23.555444 7219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 13:09:23.564186 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.556238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.0.98.62:45001)
0926 13:09:23.556289 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.556290 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.556299 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:23.556353 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.556353 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.556361 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.556405 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.556932 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.557708 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.558595 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.558794 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.558877 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.561320 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.561323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.561323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.561332 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.562724 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.563162 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.563163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.563163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.563215 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.563452 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.563453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.563453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.563596 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.563650 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.564012 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.564094 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 13:09:23.564378 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.556045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33080)
0926 13:09:23.556195 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.556233 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.556290 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.556434 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.556437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.556758 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.556827 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.556828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.557861 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.557861 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.558501 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.558503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.558910 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.558911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.558982 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.559504 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.559514 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.561232 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.562799 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.562804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.562806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.563096 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.563271 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.563271 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.563272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.563389 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.563657 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.563661 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.563867 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.563980 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.564131 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260926 13:09:23.564604 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.554022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42944 (local address 127.0.98.60:35631)
0926 13:09:23.554177 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.554186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.554215 (+ 29us) server_negotiation.cc:407] Connection header received
0926 13:09:23.554692 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.554702 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.554768 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.554970 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.555542 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.556796 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.557893 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.558373 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.558477 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.561685 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.561692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.561692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.561698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.563237 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.563577 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.563580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.563581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.563627 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.563789 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.563790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.563790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.563935 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.564056 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.564127 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.564183 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260926 13:09:23.564978 7219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 13:09:23.570786 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.565214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45555 (local address 127.0.0.1:45766)
0926 13:09:23.565245 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.565254 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.565287 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.565429 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.565429 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.565533 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.565580 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.565582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.566197 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.566198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.566725 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.566727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.567067 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.567068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.567159 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.567533 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.567538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.568571 (+ 1033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.570033 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.570034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.570034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.570182 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.570356 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.570357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.570357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.570403 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.570562 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.570562 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.570597 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.570640 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.570709 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 13:09:23.570958 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.565306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.0.98.61:45555)
0926 13:09:23.565346 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.565348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.565354 (+ 6us) server_negotiation.cc:407] Connection header received
0926 13:09:23.565374 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.565375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.565383 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.565418 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.565622 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.566150 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.566782 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.567020 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.567064 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.568660 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.568662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.568662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.568668 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.569951 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.570232 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.570233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.570233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.570278 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.570437 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.570437 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.570437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.570529 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.570570 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.570738 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.570873 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:09:23.571349 7219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 13:09:23.577041 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.571639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.0.98.60:35631)
0926 13:09:23.571734 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.571736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.571742 (+ 6us) server_negotiation.cc:407] Connection header received
0926 13:09:23.571758 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.571758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.571765 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.571806 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.571998 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.572458 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.573054 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.573294 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.573358 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.574881 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.574882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.574882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.574888 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.576051 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.576321 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.576322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.576323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.576371 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.576519 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.576519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.576520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.576620 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.576675 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.576824 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.576925 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":80}
I20260926 13:09:23.577083 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.571578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35631 (local address 127.0.0.1:42950)
0926 13:09:23.571619 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.571630 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.571669 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.571809 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.571810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.571920 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.571950 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.571951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.572508 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.572509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.572959 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.572960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.573365 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.573365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.573402 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.573744 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.573748 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.574787 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.576118 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.576118 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.576119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.576267 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.576420 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.576420 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.576421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.576470 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.576673 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.576674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.576710 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.576801 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.576967 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":86}
I20260926 13:09:23.578389 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: Bootstrap starting.
I20260926 13:09:23.579245 7219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:23.579908 7219 log.cc:824] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:23.580628 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: No bootstrap required, opened a new log
I20260926 13:09:23.581993 7219 raft_consensus.cc:348] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } }
I20260926 13:09:23.582123 7219 raft_consensus.cc:374] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:23.582155 7219 raft_consensus.cc:729] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea95093e5a546bda5ade0b7aa86c648, State: Initialized, Role: FOLLOWER
I20260926 13:09:23.582280 7219 consensus_queue.cc:260] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } }
I20260926 13:09:23.582525 7224 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } } }
I20260926 13:09:23.582578 7219 sys_catalog.cc:564] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:23.582599 7224 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 13:09:23.584106 7235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:09:23.584175 7235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:09:23.586524 7151 master_runner.cc:428] Detected that this master dea95093e5a546bda5ade0b7aa86c648 is joining an existing cluster
I20260926 13:09:23.586565 7151 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:35631
I20260926 13:09:23.594308 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.587623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45555 (local address 127.0.0.1:45782)
0926 13:09:23.587871 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.587881 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.587953 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.588115 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.588116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.588376 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.588461 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.588463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.589402 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.589403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.590107 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.590109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.590440 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.590441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.590510 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.591005 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.591010 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.591352 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.593137 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.593138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.593139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.593358 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.593593 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.593593 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.593594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.593654 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.593986 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.593987 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.594032 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.594091 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.594181 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":38,"thread_start_us":290,"threads_started":1}
I20260926 13:09:23.594302 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.587673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.0.98.61:45555)
0926 13:09:23.587796 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.587798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.587930 (+ 132us) server_negotiation.cc:407] Connection header received
0926 13:09:23.588018 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.588019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.588031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.588059 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.588559 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.589315 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.590180 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.590383 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.590436 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.591427 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.591428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.591429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.591436 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.593056 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.593439 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.593440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.593441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.593523 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.593726 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.593726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.593727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.593901 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.593981 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.594135 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.594212 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 13:09:23.595067 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.587626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.0.98.62:45001)
0926 13:09:23.587679 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.587681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.587916 (+ 235us) server_negotiation.cc:407] Connection header received
0926 13:09:23.588014 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.588015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.588026 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.588056 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.588531 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.589339 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.590516 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.590766 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.590820 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.592363 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.592365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.592365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.592372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.593732 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.594040 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.594042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.594042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.594100 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.594294 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.594295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.594295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.594416 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.594458 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.594892 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.594973 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:09:23.595383 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.587544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33086)
0926 13:09:23.587853 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.587864 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.587940 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.588117 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.588118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.588406 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.588463 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.588464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.589416 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.589417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.590407 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.590410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.591419 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.591420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.591494 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.591897 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.591901 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.592247 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.593791 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.593792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.593792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.593945 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.594169 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.594170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.594170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.594227 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.594689 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.594689 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.594732 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.595183 (+ 451us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.595285 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":40,"thread_start_us":74,"threads_started":1}
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428162, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 13:09:23.605719 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.597170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35631 (local address 127.0.0.1:42962)
0926 13:09:23.597363 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.597373 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.597413 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.597543 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.597543 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.597659 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.597694 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.597696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.598608 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.598609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.599139 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.599140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.599567 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.599567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.599608 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.599980 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.599984 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.601363 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.604694 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.604695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.604696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.604882 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.605064 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.605065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.605065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.605125 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.605408 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.605409 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.605457 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.605515 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.605616 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260926 13:09:23.605729 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.597222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.0.98.60:35631)
0926 13:09:23.597361 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.597363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.597404 (+ 41us) server_negotiation.cc:407] Connection header received
0926 13:09:23.597463 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.597463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.597473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.597541 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.597745 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.598531 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.599200 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.599509 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.599571 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.601444 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.601445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.601445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.601452 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.604604 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.604951 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.604953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.604953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.605013 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.605197 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.605199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.605199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.605338 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.605409 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.605548 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.605616 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260926 13:09:23.606216 6500 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:35631: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 35631
}
attrs {
promote: true
}
}
dest_uuid: "f40517c7135f4dd6b4f86d931669cc3f"
cas_config_opid_index: 5
I20260926 13:09:23.606462 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } }
I20260926 13:09:23.606770 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER dea95093e5a546bda5ade0b7aa86c648 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:23.607705 7059 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:35631
I20260926 13:09:23.607782 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:23.607848 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:23.608070 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.587737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35631 (local address 127.0.0.1:42954)
0926 13:09:23.587962 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.587970 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.588022 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.588159 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.588159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.603038 (+ 14879us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.603099 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.603101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.603668 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.603668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.604238 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.604241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.604557 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.604558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.604610 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.605095 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.605099 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.605409 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.607077 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.607078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.607078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.607255 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.607435 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.607435 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.607436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.607496 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.607751 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.607751 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.607793 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.607861 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.607963 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":63,"threads_started":1}
I20260926 13:09:23.608464 7193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:23.609738 7193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 FOLLOWER]: Refusing update from remote peer f40517c7135f4dd6b4f86d931669cc3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 13:09:23.610024 7151 master.cc:537] Master@127.0.98.60:35631 shutting down...
I20260926 13:09:23.610009 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:23.608465 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.587839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.0.98.60:35631)
0926 13:09:23.587870 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.587871 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.588006 (+ 135us) server_negotiation.cc:407] Connection header received
0926 13:09:23.588073 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.588074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.588081 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.588104 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.603171 (+ 15067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.603606 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.604307 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.604500 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.604553 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.605468 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.605469 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.605469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.605475 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.606995 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.607319 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.607320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.607320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.607375 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.607553 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.607554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.607554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.607672 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.607729 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.607892 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.608050 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":18}
W20260926 13:09:23.611008 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f -> Peer dea95093e5a546bda5ade0b7aa86c648 (127.0.98.60:35631): Couldn't send request to peer dea95093e5a546bda5ade0b7aa86c648. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 13:09:23.612098 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.608079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45555 (local address 127.0.0.1:45784)
0926 13:09:23.608114 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.608126 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.608164 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.608413 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.608414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.608416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:23.608456 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.608458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.609304 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.609305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.610964 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.610967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.611833 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.611834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.611915 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.611938 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.611986 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.612030 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 13:09:23.612396 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.608192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.0.98.61:45555)
0926 13:09:23.608236 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.608238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.608247 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:23.608270 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.608271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.608296 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.608357 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:23.608518 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.609213 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.611059 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.611658 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.611725 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.611800 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.612186 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.612228 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:09:23.612536 7151 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:23.612623 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 LEARNER]: Refusing update from remote peer f40517c7135f4dd6b4f86d931669cc3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 13:09:23.612627 7151 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:23.612689 7151 tablet_replica.cc:330] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: stopping tablet replica
I20260926 13:09:23.612860 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 13:09:23.613430 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER dea95093e5a546bda5ade0b7aa86c648 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:23.614522 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:23.614511 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f40517c7135f4dd6b4f86d931669cc3f. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: true } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:23.614632 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: This master's current role is: LEARNER
I20260926 13:09:23.614643 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: This master's current role is: LEARNER
I20260926 13:09:23.615038 7058 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f: attempting to promote NON_VOTER c54404f73c1e40229892ba9529e9a003 to VOTER
I20260926 13:09:23.615242 7058 consensus_queue.cc:237] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } }
I20260926 13:09:23.615722 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.615618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35631 (local address 127.0.0.1:42986)
0926 13:09:23.615663 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.615697 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 13:09:23.615808 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 LEARNER]: Refusing update from remote peer f40517c7135f4dd6b4f86d931669cc3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 13:09:23.615984 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 13:09:23.616073 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.615996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35631 (local address 127.0.0.1:43000)
0926 13:09:23.616028 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.616047 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 13:09:23.616133 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f -> Peer dea95093e5a546bda5ade0b7aa86c648 (127.0.98.60:35631): Couldn't send request to peer dea95093e5a546bda5ade0b7aa86c648. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 13:09:23.616552 7247 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c54404f73c1e40229892ba9529e9a003 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:23.616698 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c54404f73c1e40229892ba9529e9a003 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:23.617430 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.617290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35631 (local address 127.0.0.1:43002)
0926 13:09:23.617372 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.617402 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 13:09:23.617512 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:23.617584 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:09:23.617631 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:23.617708 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:23.617774 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:23.617817 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:23.626009 7151 master.cc:552] Master@127.0.98.60:35631 shutdown complete.
I20260926 13:09:23.626058 7151 master_runner.cc:305] Clearing existing system tablet
I20260926 13:09:23.626171 7151 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:23.626216 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:23.626773 7151 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:23.627313 7250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.627461 7151 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:23.627514 7151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.627538 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/wal
uuid: "dea95093e5a546bda5ade0b7aa86c648"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "1a588826284390f620f2bcd0a0d7ca27"
server_key_iv: "9d886d517ecd15bf181983f8427000d6"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.628260 7151 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:23.629482 7151 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 13:09:23.629550 7151 log.cc:1196] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 13:09:23.629710 7151 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: Deleting consensus metadata
I20260926 13:09:23.629815 7151 master_runner.cc:315] Copying system tablet from 127.0.98.62:45001
I20260926 13:09:23.630429 7151 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45001
I20260926 13:09:23.635850 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.630790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.0.98.62:45001)
0926 13:09:23.630865 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.630867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.630881 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:09:23.630948 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.630949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.630958 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.631002 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.631302 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.631829 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.632528 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.632716 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.632769 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.633674 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.633675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.633675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.633683 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.634914 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.635190 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.635191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.635192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.635240 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.635398 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.635398 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.635399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.635500 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.635555 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.635682 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.635757 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:09:23.635874 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.630700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33088)
0926 13:09:23.630801 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.630812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.630897 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.631004 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.631005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.631172 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.631235 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.631236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.631901 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.631902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.632456 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.632457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.632773 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.632774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.632843 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.633325 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.633328 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.633597 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.634975 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.634976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.634977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.635134 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.635289 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.635290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.635290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.635345 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.635550 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.635551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.635595 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.635653 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.635756 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":67,"threads_started":1}
I20260926 13:09:23.636219 6520 tablet_copy_service.cc:140] P f40517c7135f4dd6b4f86d931669cc3f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dea95093e5a546bda5ade0b7aa86c648 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33088)
I20260926 13:09:23.636272 6520 tablet_copy_service.cc:161] P f40517c7135f4dd6b4f86d931669cc3f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dea95093e5a546bda5ade0b7aa86c648 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33088: session id = dea95093e5a546bda5ade0b7aa86c648-00000000000000000000000000000000
I20260926 13:09:23.636790 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 13:09:23.637240 7151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 13:09:23.638118 7151 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: tablet copy: Starting download of 0 data blocks...
I20260926 13:09:23.638238 7151 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: tablet copy: Starting download of 1 WAL segments...
I20260926 13:09:23.638983 7151 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 13:09:23.639716 6520 tablet_copy_service.cc:342] P f40517c7135f4dd6b4f86d931669cc3f: Request end of tablet copy session dea95093e5a546bda5ade0b7aa86c648-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33088
I20260926 13:09:23.639767 6520 tablet_copy_service.cc:433] P f40517c7135f4dd6b4f86d931669cc3f: ending tablet copy session dea95093e5a546bda5ade0b7aa86c648-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dea95093e5a546bda5ade0b7aa86c648
I20260926 13:09:23.641865 7151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:23.643065 7257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:23.643076 7260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:23.643301 7151 server_base.cc:1034] running on GCE node
W20260926 13:09:23.643235 7258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:23.643450 7151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:23.643576 7151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:23.644690 7151 hybrid_clock.cc:648] HybridClock initialized: now 1790428163644683 us; error 22 us; skew 500 ppm
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:23.646025 7151 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 13:09:23.646268 7151 webserver.cc:460] Webserver started at http://127.0.98.60:34801/ using document root <none> and password file <none>
I20260926 13:09:23.646375 7151 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:23.646407 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:23.647063 7151 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:09:23.647559 7267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.647678 7151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.647710 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/wal
uuid: "dea95093e5a546bda5ade0b7aa86c648"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "1a588826284390f620f2bcd0a0d7ca27"
server_key_iv: "9d886d517ecd15bf181983f8427000d6"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.647764 7151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:23.664323 7151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:23.666013 7151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:23.669377 7151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35631
I20260926 13:09:23.669392 7319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35631 every 8 connection(s)
I20260926 13:09:23.669705 7151 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
I20260926 13:09:23.670584 7320 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:09:23.671187 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: Bootstrap starting.
I20260926 13:09:23.674276 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:09:23.674487 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: Bootstrap complete.
I20260926 13:09:23.674671 7320 raft_consensus.cc:348] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } }
I20260926 13:09:23.674767 7320 raft_consensus.cc:729] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dea95093e5a546bda5ade0b7aa86c648, State: Initialized, Role: LEARNER
I20260926 13:09:23.674815 7320 consensus_queue.cc:260] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } }
I20260926 13:09:23.675043 7320 sys_catalog.cc:564] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:23.675055 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:23.675189 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: This master's current role is: LEARNER
I20260926 13:09:23.676995 7334 catalog_manager.cc:1259] Loaded cluster ID: 412444a8b1f04915b5763a001a60229f
I20260926 13:09:23.677404 7334 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: loading cluster ID for follower catalog manager: success
I20260926 13:09:23.678117 7334 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: acquiring CA information for follower catalog manager: success
I20260926 13:09:23.678370 7334 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:09:23.722672 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.716685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.0.98.60:35631)
0926 13:09:23.716828 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.716830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.716838 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:23.716864 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.716865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.716877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.716940 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.717508 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.718161 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.718830 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.719067 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.719121 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.720312 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.720313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.720314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.720321 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.721678 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.721984 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.721985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.721985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.722036 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.722199 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.722200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.722200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.722304 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.722396 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.722487 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.722578 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260926 13:09:23.723264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 6562
I20260926 13:09:23.728293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40611
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45001
--webserver_port=45219
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45001,127.0.98.61:45555
--tserver_master_addrs=127.0.98.62:45001,127.0.98.61:45555,127.0.98.60:35631 with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:23.808624 7336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:23.808851 7336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:23.808874 7336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:23.810020 7336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:23.810063 7336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:23.810089 7336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:23.811542 7336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45219
--tserver_master_addrs=127.0.98.62:45001,127.0.98.61:45555,127.0.98.60:35631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:23.811740 7336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:23.811941 7336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:23.816870 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:23.816923 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:23.816921 7341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:23.817312 7336 server_base.cc:1034] running on GCE node
I20260926 13:09:23.817510 7336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:23.817713 7336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:23.818917 7336 hybrid_clock.cc:648] HybridClock initialized: now 1790428163818898 us; error 31 us; skew 500 ppm
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:23.821326 7336 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 13:09:23.821784 7336 webserver.cc:460] Webserver started at http://127.0.98.1:45219/ using document root <none> and password file <none>
I20260926 13:09:23.822000 7336 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:23.822060 7336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:23.823490 7336 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:23.824070 7351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.824221 7336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:23.824301 7336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "2caca43a229c408aa688a866f07250b6"
format_stamp: "Formatted at 2026-09-26 13:09:22 on dist-test-slave-2dw5"
server_key: "947a430f3704185150cb5ca9cbefabc9"
server_key_iv: "5dedb8925aea45f6a5992c72abfdc699"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.824669 7336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:23.847885 7336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:23.849889 7336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:23.850032 7336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:23.850263 7336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:23.850557 7336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:23.850589 7336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.850618 7336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:23.850641 7336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.856702 7336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40611
I20260926 13:09:23.856748 7464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40611 every 8 connection(s)
I20260926 13:09:23.857077 7336 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:09:23.863216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 7336
I20260926 13:09:23.863369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 6696
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:23.869624 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.857825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50513 (local address 127.0.98.61:45555)
0926 13:09:23.857883 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.857884 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.858188 (+ 304us) server_negotiation.cc:407] Connection header received
0926 13:09:23.859334 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.859335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.859346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.859405 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.860147 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.860921 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.862109 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.862433 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.862488 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.865750 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.865751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.865752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.865759 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.867806 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.868515 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.868516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.868517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.868580 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.868860 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.868861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.868861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.869009 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.869052 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.869446 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.869530 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 13:09:23.869668 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.857788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53095 (local address 127.0.98.62:45001)
0926 13:09:23.857842 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.857844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.858094 (+ 250us) server_negotiation.cc:407] Connection header received
0926 13:09:23.859190 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.859192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.859207 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.859273 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.860148 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.860924 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.862049 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.862280 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.862312 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.865813 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.865815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.865815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.865822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.868022 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.868536 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.868536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.868537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.868617 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.868890 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.868891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.868891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.869065 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.869129 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.869471 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.869564 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:09:23.870102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44653
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45001
--webserver_port=37413
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45001,127.0.98.61:45555
--tserver_master_addrs=127.0.98.62:45001,127.0.98.61:45555,127.0.98.60:35631 with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:09:23.870309 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.857800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.1:53095)
0926 13:09:23.857965 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.858001 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.859046 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.859330 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.859333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.859831 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.860044 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.860050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.861013 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.861014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.861938 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.861941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.862387 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.862387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.862529 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.863401 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.863406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.865712 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.868100 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.868101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.868101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.868427 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.868747 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.868747 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.868748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.868795 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.869168 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.869168 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.869296 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.869805 (+ 509us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.870067 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":36,"thread_start_us":103,"threads_started":1}
I20260926 13:09:23.870895 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.857800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45555 (local address 127.0.98.1:50513)
0926 13:09:23.858104 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.858115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.859048 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.859414 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.859415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.859851 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.860044 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.860050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.861013 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.861014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.862007 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.862009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.862537 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.862538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.862593 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.863279 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.863289 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.865660 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.867928 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.867962 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.867966 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.868427 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.868651 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.868652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.868652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.868785 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.869099 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.869105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.869290 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.870134 (+ 844us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.870771 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":68,"thread_start_us":98,"threads_started":1}
I20260926 13:09:23.870895 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45001
I20260926 13:09:23.871025 7466 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:23.871276 7466 heartbeater.cc:511] Master 127.0.98.62:45001 requested a full tablet report, sending...
I20260926 13:09:23.871577 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.858265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42741 (local address 127.0.98.60:35631)
0926 13:09:23.858307 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.858309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.858319 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:23.859192 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.859193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.859208 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.859241 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.860178 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.860770 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.862039 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.862358 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.862440 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.865839 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:23.865841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:23.865841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:23.865848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:23.870094 (+ 4246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.870437 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.870439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.870439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.870505 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:23.870976 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:23.870977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:23.870978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:23.871118 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:23.871200 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:23.871401 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 13:09:23.871492 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:09:23.871713 6500 ts_manager.cc:194] Re-registered known tserver with Master: 2caca43a229c408aa688a866f07250b6 (127.0.98.1:40611)
I20260926 13:09:23.871958 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.858099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35631 (local address 127.0.98.1:42741)
0926 13:09:23.858211 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.858220 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.859068 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.859332 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.859335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.859813 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.860042 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.860047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.860898 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.860899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.861885 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.861888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.862450 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.862450 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.862529 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.863983 (+ 1454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.863988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.865420 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:23.870186 (+ 4766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.870187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.870187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.870374 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.870654 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:23.870655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:23.870656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:23.870782 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:23.871237 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:23.871238 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:23.871283 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 13:09:23.871658 (+ 375us) client_negotiation.cc:240] Negotiation successful
0926 13:09:23.871764 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":5,"thread_start_us":64,"threads_started":1}
I20260926 13:09:23.872202 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53095
I20260926 13:09:23.872311 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35631
I20260926 13:09:23.872362 7465 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:23.872426 7465 heartbeater.cc:511] Master 127.0.98.60:35631 requested a full tablet report, sending...
I20260926 13:09:23.872535 7467 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45555
I20260926 13:09:23.872570 7467 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:23.872622 7467 heartbeater.cc:511] Master 127.0.98.61:45555 requested a full tablet report, sending...
I20260926 13:09:23.872788 7284 ts_manager.cc:194] Registered new tserver with Master: 2caca43a229c408aa688a866f07250b6 (127.0.98.1:40611)
I20260926 13:09:23.873366 7097 ts_manager.cc:194] Registered new tserver with Master: 2caca43a229c408aa688a866f07250b6 (127.0.98.1:40611)
W20260926 13:09:23.953009 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:23.953220 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:23.953243 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:23.954401 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:23.954448 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:23.954474 7472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:23.956032 7472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37413
--tserver_master_addrs=127.0.98.62:45001,127.0.98.61:45555,127.0.98.60:35631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:23.956262 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:23.956483 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:23.961447 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:23.961460 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:23.961457 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:23.961660 7472 server_base.cc:1034] running on GCE node
I20260926 13:09:23.962036 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:23.962288 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:23.963438 7472 hybrid_clock.cc:648] HybridClock initialized: now 1790428163963405 us; error 48 us; skew 500 ppm
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:23.965943 7472 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 13:09:23.966355 7472 webserver.cc:460] Webserver started at http://127.0.98.2:37413/ using document root <none> and password file <none>
I20260926 13:09:23.966557 7472 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:23.966614 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:23.967972 7472 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:23.968511 7487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.968644 7472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:23.968720 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "7bb70baa07584f89868a7f334975c060"
format_stamp: "Formatted at 2026-09-26 13:09:22 on dist-test-slave-2dw5"
server_key: "974876946fa12b2ce4eb3cf76ea48454"
server_key_iv: "3648be212b69953bc28776c1225cf547"
server_key_version: "encryptionkey@0"
I20260926 13:09:23.969100 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:23.983028 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:23.985139 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:23.985283 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:23.985546 7472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:23.985855 7472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:23.985888 7472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.985909 7472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:23.985921 7472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:23.991542 7472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44653
I20260926 13:09:23.991580 7600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44653 every 8 connection(s)
I20260926 13:09:23.991920 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:09:23.996038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 7472
I20260926 13:09:23.996201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 6830
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 13:09:23 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:24.003302 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.992962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58235 (local address 127.0.98.60:35631)
0926 13:09:23.993005 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.993007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.993128 (+ 121us) server_negotiation.cc:407] Connection header received
0926 13:09:23.993472 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.993473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.993484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.993529 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.994385 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.995202 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.996539 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.996835 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.996870 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.000092 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:24.000094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:24.000094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:24.000101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:24.001725 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.002175 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.002176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.002176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.002232 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.002530 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.002531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.002531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.002670 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:24.002708 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.003110 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.003199 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:09:24.003540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35143
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45001
--webserver_port=39047
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45001,127.0.98.61:45555
--tserver_master_addrs=127.0.98.62:45001,127.0.98.61:45555,127.0.98.60:35631 with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:09:24.003708 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.992577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38103 (local address 127.0.98.61:45555)
0926 13:09:23.992660 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.992661 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.992905 (+ 244us) server_negotiation.cc:407] Connection header received
0926 13:09:23.993497 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.993498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.993510 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.993540 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.994416 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.995203 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.996398 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.996674 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.996714 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.000523 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:24.000524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:24.000524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:24.000532 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:24.002532 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.002944 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.002946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.002946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.003005 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.003181 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.003182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.003182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.003298 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:24.003352 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.003555 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.003629 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:09:24.003829 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.992876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35631 (local address 127.0.98.2:58235)
0926 13:09:23.993055 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.993064 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.993395 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.993856 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.993857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.994084 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.994299 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.994303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.995661 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.995662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.996460 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.996462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.996936 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.996936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.996987 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.997603 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.997608 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:23.999994 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:24.001804 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:24.001812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:24.001814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:24.002089 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:24.002301 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:24.002301 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:24.002302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:24.002452 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:24.002751 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:24.002757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:24.002962 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.003412 (+ 450us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.003600 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":4,"thread_start_us":53,"threads_started":1}
I20260926 13:09:24.004015 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.992606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45555 (local address 127.0.98.2:38103)
0926 13:09:23.992768 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.992815 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.993385 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.993615 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.993619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.994115 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.994299 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.994305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.995279 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.995279 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.996292 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.996295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.996803 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.996804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.996959 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.997588 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.997598 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:24.000422 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:24.002619 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:24.002620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:24.002621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:24.002810 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:24.003070 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:24.003071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:24.003071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:24.003124 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:24.003407 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:24.003408 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:24.003451 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.003813 (+ 362us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.003919 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":49,"spinlock_wait_cycles":384,"thread_start_us":86,"threads_started":1}
I20260926 13:09:24.004385 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45555
I20260926 13:09:24.004508 7603 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:24.004720 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.992576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59887 (local address 127.0.98.62:45001)
0926 13:09:23.992624 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:23.992626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:23.992905 (+ 279us) server_negotiation.cc:407] Connection header received
0926 13:09:23.993497 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:23.993498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:23.993511 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:23.993540 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:23.994407 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.994992 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.996449 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.996731 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.996769 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.000964 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:24.000966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:24.000966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:24.000974 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:24.002663 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.003071 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.003072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.003072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.003129 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.003308 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.003309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.003309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.003440 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:24.003492 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.004555 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.004642 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:09:24.004762 7603 heartbeater.cc:511] Master 127.0.98.61:45555 requested a full tablet report, sending...
I20260926 13:09:24.005124 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35631
I20260926 13:09:24.005175 7601 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:24.005230 7601 heartbeater.cc:511] Master 127.0.98.60:35631 requested a full tablet report, sending...
I20260926 13:09:24.005249 7097 ts_manager.cc:194] Registered new tserver with Master: 7bb70baa07584f89868a7f334975c060 (127.0.98.2:44653)
I20260926 13:09:24.005255 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:23.992606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.2:59887)
0926 13:09:23.992769 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:23.992815 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:23.993385 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:23.993616 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:23.993619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:23.994117 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:23.994299 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.994305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.995093 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.995094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.996354 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:23.996356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:23.996843 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:23.996844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:23.996959 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:23.997697 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:23.997701 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:24.000887 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:24.002742 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:24.002743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:24.002743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:24.002985 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:24.003196 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:24.003196 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:24.003196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:24.003248 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:24.004379 (+ 1131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:24.004380 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:24.004436 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.004836 (+ 400us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.004951 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":7,"thread_start_us":85,"threads_started":1}
I20260926 13:09:24.005594 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45001
I20260926 13:09:24.005586 7284 ts_manager.cc:194] Registered new tserver with Master: 7bb70baa07584f89868a7f334975c060 (127.0.98.2:44653)
I20260926 13:09:24.005638 7602 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:24.005689 7602 heartbeater.cc:511] Master 127.0.98.62:45001 requested a full tablet report, sending...
I20260926 13:09:24.005936 6500 ts_manager.cc:194] Re-registered known tserver with Master: 7bb70baa07584f89868a7f334975c060 (127.0.98.2:44653)
I20260926 13:09:24.006316 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59887
I20260926 13:09:24.030560 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.027574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.0.98.60:35631)
0926 13:09:24.027627 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.027629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.027641 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:09:24.027667 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.027668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.027686 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.027752 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:24.027889 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.028431 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.029813 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.030252 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.030319 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.030375 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.030468 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.030496 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:09:24.030589 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.027502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35631 (local address 127.0.0.1:43028)
0926 13:09:24.027559 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:24.027571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:24.027612 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:24.027768 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:24.027768 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:24.027770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:24.027826 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.027828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.028517 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.028518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.029709 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.029714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.030335 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.030336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.030402 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.030407 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.030460 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.030511 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 13:09:24.030951 7294 raft_consensus.cc:1203] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 13:09:24.031101 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f40517c7135f4dd6b4f86d931669cc3f. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:24.031219 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: This master's current role is: LEARNER
W20260926 13:09:24.086169 7608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:24.086433 7608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:24.086467 7608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:24.087697 7608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:24.087745 7608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:24.087781 7608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:24.089521 7608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39047
--tserver_master_addrs=127.0.98.62:45001,127.0.98.61:45555,127.0.98.60:35631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:24.089771 7608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:24.089982 7608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:24.094895 7614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:24.094870 7613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:24.095062 7608 server_base.cc:1034] running on GCE node
W20260926 13:09:24.095002 7616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:24.096359 7608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:24.096616 7608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:24.097760 7608 hybrid_clock.cc:648] HybridClock initialized: now 1790428164097745 us; error 32 us; skew 500 ppm
Sep 26 13:09:24 dist-test-slave-2dw5 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428164, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:24.100234 7608 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 13:09:24.100718 7608 webserver.cc:460] Webserver started at http://127.0.98.3:39047/ using document root <none> and password file <none>
I20260926 13:09:24.100926 7608 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:24.100980 7608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:24.102264 7608 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:24.102823 7623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:24.102998 7608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:24.103080 7608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "c8137509b54b41d78e847082f20b7fb5"
format_stamp: "Formatted at 2026-09-26 13:09:23 on dist-test-slave-2dw5"
server_key: "955cb7a087ee056ba174cd6fcbdb23ce"
server_key_iv: "88ef78a5272a2d24544bb0d3d9313cc3"
server_key_version: "encryptionkey@0"
I20260926 13:09:24.103451 7608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:24.117422 7608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:24.119639 7608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:24.119778 7608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:24.119972 7608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:24.120252 7608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:24.120285 7608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:24.120312 7608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:24.120335 7608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:24.125975 7608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35143
I20260926 13:09:24.126025 7736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35143 every 8 connection(s)
I20260926 13:09:24.126359 7608 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:24.129226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 7608
Sep 26 13:09:24 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428164, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 13:09:24 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428164, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 13:09:24 dist-test-slave-2dw5 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428164, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 13:09:24.139252 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.127423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57597 (local address 127.0.98.61:45555)
0926 13:09:24.127471 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.127473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.127656 (+ 183us) server_negotiation.cc:407] Connection header received
0926 13:09:24.128630 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.128631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.128641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.128684 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:24.129650 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.130420 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.132928 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.133209 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.133264 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.136054 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:24.136055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:24.136055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:24.136062 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:24.137639 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.138142 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.138143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.138144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.138203 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.138495 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.138496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.138496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.138638 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:24.138694 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.139070 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.139150 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:09:24.139617 7335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.127078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55297 (local address 127.0.98.60:35631)
0926 13:09:24.127127 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.127129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.127423 (+ 294us) server_negotiation.cc:407] Connection header received
0926 13:09:24.128599 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.128601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.128614 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.128644 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:24.129651 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.130423 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.132953 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.133253 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.133295 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.137135 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:24.137137 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:24.137137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:24.137144 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:24.138480 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.138802 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.138803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.138804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.138886 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.139087 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.139087 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.139088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.139208 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:24.139262 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.139461 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.139534 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:09:24.140230 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.127104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45555 (local address 127.0.98.3:57597)
0926 13:09:24.127563 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:24.127573 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:24.128549 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:24.128741 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:24.128742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:24.129213 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:24.129536 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.129544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.130517 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.130517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.132809 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.132811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.133354 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.133356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.133415 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.133818 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:24.133822 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:24.135966 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:24.137729 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:24.137739 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:24.137742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:24.138064 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:24.138285 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:24.138286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:24.138286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:24.138431 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:24.138700 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:24.138708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:24.138914 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.139628 (+ 714us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.139953 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":1316,"thread_start_us":128,"threads_started":1}
I20260926 13:09:24.141017 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45555
I20260926 13:09:24.141119 7739 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:24.141265 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.130597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.0.98.61:45555)
0926 13:09:24.131053 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.131055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.131253 (+ 198us) server_negotiation.cc:407] Connection header received
0926 13:09:24.131275 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.131276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.131287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.131344 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:24.132099 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.133252 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.134999 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.135274 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.135349 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.136768 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:24.136770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:24.136770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:24.136778 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:24.139518 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.140348 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.140349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.140350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.140422 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.140626 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.140627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.140628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.140759 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:24.140850 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.141084 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.141166 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":53,"threads_started":1}
I20260926 13:09:24.141312 7739 heartbeater.cc:511] Master 127.0.98.61:45555 requested a full tablet report, sending...
I20260926 13:09:24.141644 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.131019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.0.98.62:45001)
0926 13:09:24.131664 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.131666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.131673 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:09:24.131693 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.131693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.131704 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.132369 (+ 665us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:24.132402 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.133080 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.134102 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.134378 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.134725 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.135848 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:24.135850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:24.135850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:24.135857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:24.140056 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.140403 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.140404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.140404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.140458 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.140653 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.140654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.140654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.140784 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:24.141426 (+ 642us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.141464 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.141549 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":60,"threads_started":1}
I20260926 13:09:24.141901 7097 ts_manager.cc:194] Registered new tserver with Master: c8137509b54b41d78e847082f20b7fb5 (127.0.98.3:35143)
I20260926 13:09:24.142150 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.127434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.3:60231)
0926 13:09:24.127663 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:24.127673 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:24.128477 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:24.128723 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:24.128726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:24.129243 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:24.129536 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.129544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.130506 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.130507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.132000 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.132003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.132506 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.132507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.132648 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.133375 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:24.133386 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:24.135806 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:24.140005 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:24.140006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:24.140007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:24.140247 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:24.141437 (+ 1190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:24.141438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:24.141438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:24.141500 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:24.141824 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:24.141825 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:24.141880 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.141940 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.142047 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":82,"thread_start_us":60,"threads_started":1}
I20260926 13:09:24.140236 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.127104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35631 (local address 127.0.98.3:55297)
0926 13:09:24.127300 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:24.127336 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:24.128471 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:24.128722 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:24.128727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:24.129236 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:24.129536 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.129544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.130517 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.130517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.132000 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.132003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.133612 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.133612 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.133672 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.134389 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:24.134394 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:24.137036 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:24.138548 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:24.138548 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:24.138549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:24.138733 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:24.138959 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:24.138960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:24.138960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:24.139019 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:24.139311 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:24.139312 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:24.139352 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.139696 (+ 344us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.139882 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":270,"thread_start_us":132,"threads_started":1}
I20260926 13:09:24.142194 6547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.127571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60231 (local address 127.0.98.62:45001)
0926 13:09:24.127628 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.127630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.127722 (+ 92us) server_negotiation.cc:407] Connection header received
0926 13:09:24.128578 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.128580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.128594 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.128633 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:24.129660 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.130371 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.132099 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.132385 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.132423 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.135897 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:24.135897 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:24.135898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:24.135903 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:24.138451 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.141184 (+ 2733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.141185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.141185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.141242 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.141574 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.141575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.141575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.141689 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:24.141740 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.142035 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.142118 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:09:24.142944 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35631
I20260926 13:09:24.142997 7737 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:24.143064 7737 heartbeater.cc:511] Master 127.0.98.60:35631 requested a full tablet report, sending...
I20260926 13:09:24.143069 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45001
I20260926 13:09:24.143297 7738 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:24.143360 7738 heartbeater.cc:511] Master 127.0.98.62:45001 requested a full tablet report, sending...
I20260926 13:09:24.143483 7284 ts_manager.cc:194] Registered new tserver with Master: c8137509b54b41d78e847082f20b7fb5 (127.0.98.3:35143)
I20260926 13:09:24.143570 6500 ts_manager.cc:194] Re-registered known tserver with Master: c8137509b54b41d78e847082f20b7fb5 (127.0.98.3:35143)
I20260926 13:09:24.144028 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60231
I20260926 13:09:24.144577 6500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:24.146391 6500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:24.147257 7756 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: true } } }
I20260926 13:09:24.147361 7756 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:24.148617 7756 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f: attempting to promote NON_VOTER dea95093e5a546bda5ade0b7aa86c648 to VOTER
I20260926 13:09:24.148777 7756 consensus_queue.cc:237] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } }
I20260926 13:09:24.149150 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 FOLLOWER]: Refusing update from remote peer f40517c7135f4dd6b4f86d931669cc3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 13:09:24.149217 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.131227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43030 (local address 127.0.98.60:35631)
0926 13:09:24.131442 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.131444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.134927 (+ 3483us) server_negotiation.cc:407] Connection header received
0926 13:09:24.140527 (+ 5600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.140529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.140543 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.140582 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:24.142327 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.143059 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.144095 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.144393 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.144469 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.146054 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:24.146055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:24.146056 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:24.146063 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:24.147988 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.148372 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.148374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.148374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.148438 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.148647 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.148648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.148648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.148794 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:24.148889 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.149027 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.149118 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":46,"threads_started":1}
I20260926 13:09:24.149235 7294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 LEARNER]: Refusing update from remote peer f40517c7135f4dd6b4f86d931669cc3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 13:09:24.149334 7755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 13:09:24.149411 7247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 13:09:24.150254 7247 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, dea95093e5a546bda5ade0b7aa86c648 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } } }
I20260926 13:09:24.150508 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, dea95093e5a546bda5ade0b7aa86c648 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } } }
I20260926 13:09:24.151196 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } } }
I20260926 13:09:24.151268 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:24.151316 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } } }
I20260926 13:09:24.151355 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:24.151396 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } } }
I20260926 13:09:24.151428 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:24.151638 7294 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, dea95093e5a546bda5ade0b7aa86c648 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } } }
I20260926 13:09:24.152474 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } } }
I20260926 13:09:24.152561 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:09:24.153018 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f40517c7135f4dd6b4f86d931669cc3f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } } }
I20260926 13:09:24.153122 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:09:24.158257 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.154395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.0.98.3:35143)
0926 13:09:24.154713 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.154718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.154729 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:24.154770 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.154772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.154788 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.154892 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:24.155032 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.156080 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.157271 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.157905 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.158007 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.158088 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.158140 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.158176 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":100,"threads_started":1}
I20260926 13:09:24.158243 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.154270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35143 (local address 127.0.0.1:57204)
0926 13:09:24.154315 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:24.154326 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:24.154361 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:24.154903 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:24.154904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:24.154907 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:24.154963 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.154965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.156153 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.156154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.157215 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.157217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.158009 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.158010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.158069 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.158073 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.158122 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.158166 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 13:09:24.158507 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.154325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40611 (local address 127.0.0.1:49230)
0926 13:09:24.154463 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:24.154471 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:24.154507 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:24.155087 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:24.155088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:24.155089 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:24.155118 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.155118 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.155944 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.155945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.157110 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.157112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.158277 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.158278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.158339 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.158343 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.158379 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.158433 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":26,"thread_start_us":110,"threads_started":1}
I20260926 13:09:24.158550 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.154562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.0.98.1:40611)
0926 13:09:24.154955 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.154958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.154966 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:24.154998 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.155000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.155013 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.155084 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:24.155181 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.155868 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.157204 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.158181 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.158269 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.158417 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.158442 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.158481 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":79,"threads_started":1}
I20260926 13:09:24.158766 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.154325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44653 (local address 127.0.0.1:51742)
0926 13:09:24.154578 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:24.154586 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:24.154622 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:24.155225 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:24.155226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:24.155229 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:24.155568 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.155571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.156564 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.156565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.157318 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.157321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.158530 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.158530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.158582 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.158586 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.158640 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.158684 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":4,"thread_start_us":98,"threads_started":1}
I20260926 13:09:24.159040 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.154565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.0.98.2:44653)
0926 13:09:24.154732 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.154736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.154744 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:24.154778 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.154780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.154794 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.155169 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:24.155643 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.156492 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.157421 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.158086 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.158153 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.158223 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.158937 (+ 714us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.158981 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260926 13:09:24.159289 7399 tablet_service.cc:1467] Processing CreateTablet for tablet b2fdab9092be4c56ae301e105bd8cb21 (DEFAULT_TABLE table=test-table [id=f0cce960c33647aaa6e70ff33801e34d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:24.159618 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2fdab9092be4c56ae301e105bd8cb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:24.159652 7671 tablet_service.cc:1467] Processing CreateTablet for tablet b2fdab9092be4c56ae301e105bd8cb21 (DEFAULT_TABLE table=test-table [id=f0cce960c33647aaa6e70ff33801e34d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:24.159947 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2fdab9092be4c56ae301e105bd8cb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:24.162688 7771 tablet_bootstrap.cc:492] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5: Bootstrap starting.
I20260926 13:09:24.162705 7770 tablet_bootstrap.cc:492] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6: Bootstrap starting.
I20260926 13:09:24.163058 7535 tablet_service.cc:1467] Processing CreateTablet for tablet b2fdab9092be4c56ae301e105bd8cb21 (DEFAULT_TABLE table=test-table [id=f0cce960c33647aaa6e70ff33801e34d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:24.163280 7771 tablet_bootstrap.cc:654] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:24.163400 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2fdab9092be4c56ae301e105bd8cb21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:24.163393 7770 tablet_bootstrap.cc:654] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:24.163565 7771 log.cc:824] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:24.163698 7770 log.cc:824] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:24.164531 7770 tablet_bootstrap.cc:492] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6: No bootstrap required, opened a new log
I20260926 13:09:24.164609 7770 ts_tablet_manager.cc:1397] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:24.167248 7771 tablet_bootstrap.cc:492] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5: No bootstrap required, opened a new log
I20260926 13:09:24.167367 7771 ts_tablet_manager.cc:1397] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 13:09:24.167448 7774 tablet_bootstrap.cc:492] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060: Bootstrap starting.
I20260926 13:09:24.167514 7770 raft_consensus.cc:348] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } } peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } }
I20260926 13:09:24.167690 7770 raft_consensus.cc:374] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:24.167728 7770 raft_consensus.cc:729] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2caca43a229c408aa688a866f07250b6, State: Initialized, Role: FOLLOWER
I20260926 13:09:24.167847 7770 consensus_queue.cc:260] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } } peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } }
I20260926 13:09:24.168099 7770 ts_tablet_manager.cc:1428] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260926 13:09:24.168212 7774 tablet_bootstrap.cc:654] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:24.168495 7774 log.cc:824] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:24.168774 7466 heartbeater.cc:503] Master 127.0.98.62:45001 was elected leader, sending a full tablet report...
I20260926 13:09:24.169445 7774 tablet_bootstrap.cc:492] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060: No bootstrap required, opened a new log
I20260926 13:09:24.169513 7774 ts_tablet_manager.cc:1397] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:24.170109 7771 raft_consensus.cc:348] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } } peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } }
I20260926 13:09:24.170279 7771 raft_consensus.cc:374] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:24.170315 7771 raft_consensus.cc:729] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8137509b54b41d78e847082f20b7fb5, State: Initialized, Role: FOLLOWER
I20260926 13:09:24.170440 7771 consensus_queue.cc:260] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } } peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } }
I20260926 13:09:24.170702 7771 ts_tablet_manager.cc:1428] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:09:24.170744 7738 heartbeater.cc:503] Master 127.0.98.62:45001 was elected leader, sending a full tablet report...
I20260926 13:09:24.171828 7774 raft_consensus.cc:348] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } } peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } }
I20260926 13:09:24.172001 7774 raft_consensus.cc:374] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:24.172040 7774 raft_consensus.cc:729] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb70baa07584f89868a7f334975c060, State: Initialized, Role: FOLLOWER
I20260926 13:09:24.172150 7774 consensus_queue.cc:260] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } } peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } }
I20260926 13:09:24.172382 7774 ts_tablet_manager.cc:1428] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:09:24.172442 7602 heartbeater.cc:503] Master 127.0.98.62:45001 was elected leader, sending a full tablet report...
W20260926 13:09:24.243467 7604 tablet.cc:2369] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:09:24.331339 7777 raft_consensus.cc:482] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:24.331483 7777 raft_consensus.cc:504] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } } peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } }
I20260926 13:09:24.331648 7775 raft_consensus.cc:482] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:24.331753 7775 raft_consensus.cc:504] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } } peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } }
I20260926 13:09:24.332048 7775 leader_election.cc:288] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8137509b54b41d78e847082f20b7fb5 (127.0.98.3:35143), 7bb70baa07584f89868a7f334975c060 (127.0.98.2:44653)
I20260926 13:09:24.332458 7777 leader_election.cc:288] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bb70baa07584f89868a7f334975c060 (127.0.98.2:44653), 2caca43a229c408aa688a866f07250b6 (127.0.98.1:40611)
I20260926 13:09:24.335829 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.332354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40611 (local address 127.0.98.3:52405)
0926 13:09:24.332480 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:24.332489 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:24.332518 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:24.332637 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:24.332638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:24.332640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:24.332722 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.332724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.333542 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.333543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.334792 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.334794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.335591 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.335592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.335663 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.335666 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.335705 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.335745 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":6}
I20260926 13:09:24.335915 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.332213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35143 (local address 127.0.98.1:39793)
0926 13:09:24.332298 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:24.332308 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:24.332351 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:24.332491 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:24.332493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:24.332496 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:24.332591 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.332593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.333501 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.333502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.334785 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.334789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.335683 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.335684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.335752 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.335756 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.335796 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.335835 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 13:09:24.335952 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.332164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48315 (local address 127.0.98.2:44653)
0926 13:09:24.332305 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.332308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.332315 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:09:24.332343 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.332344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.332356 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.332400 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:24.332904 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.333579 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.334916 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.335556 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.335622 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.335633 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.335858 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.335896 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260926 13:09:24.336114 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.332433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52405 (local address 127.0.98.1:40611)
0926 13:09:24.332521 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.332523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.332536 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:09:24.332557 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.332557 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.332565 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.332610 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:24.332803 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.333448 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.334886 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.335481 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.335529 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.335540 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.336023 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.336058 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 13:09:24.336184 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.332250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39793 (local address 127.0.98.3:35143)
0926 13:09:24.332361 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.332362 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.332372 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:24.332392 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.332393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.332402 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.332446 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:24.332665 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.333417 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.335037 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.335596 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.335647 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.335655 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.336083 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.336126 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 13:09:24.336344 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.332076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44653 (local address 127.0.98.3:48315)
0926 13:09:24.332174 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:24.332188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:24.332234 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:24.332768 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:24.332769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:24.332782 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:24.332823 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.332824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.333652 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.333653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.334786 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.334789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.335659 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.335660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.335724 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.335726 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.335770 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.335809 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 13:09:24.336449 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fdab9092be4c56ae301e105bd8cb21" candidate_uuid: "2caca43a229c408aa688a866f07250b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8137509b54b41d78e847082f20b7fb5" is_pre_election: true
I20260926 13:09:24.336556 7691 raft_consensus.cc:2454] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2caca43a229c408aa688a866f07250b6 in term 0.
I20260926 13:09:24.336762 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fdab9092be4c56ae301e105bd8cb21" candidate_uuid: "c8137509b54b41d78e847082f20b7fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2caca43a229c408aa688a866f07250b6" is_pre_election: true
I20260926 13:09:24.336790 7353 leader_election.cc:302] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2caca43a229c408aa688a866f07250b6, c8137509b54b41d78e847082f20b7fb5; no voters:
I20260926 13:09:24.336870 7419 raft_consensus.cc:2454] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8137509b54b41d78e847082f20b7fb5 in term 0.
I20260926 13:09:24.336935 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fdab9092be4c56ae301e105bd8cb21" candidate_uuid: "c8137509b54b41d78e847082f20b7fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb70baa07584f89868a7f334975c060" is_pre_election: true
I20260926 13:09:24.337013 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.332631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32803 (local address 127.0.98.2:44653)
0926 13:09:24.333133 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.333135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.333143 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:24.333163 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.333164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.333174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.333378 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:24.333405 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.334673 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.335984 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.336545 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.336827 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.336841 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.336859 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.336895 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":105,"threads_started":1}
I20260926 13:09:24.337026 7627 leader_election.cc:302] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2caca43a229c408aa688a866f07250b6, c8137509b54b41d78e847082f20b7fb5; no voters:
I20260926 13:09:24.337116 7555 raft_consensus.cc:2454] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8137509b54b41d78e847082f20b7fb5 in term 0.
I20260926 13:09:24.337133 7777 raft_consensus.cc:2781] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:24.337177 7777 raft_consensus.cc:482] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:24.337196 7777 raft_consensus.cc:3037] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:24.337203 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.332221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44653 (local address 127.0.98.1:32803)
0926 13:09:24.332421 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:24.332441 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:24.332467 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:24.333262 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:24.333263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:24.333265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:24.333325 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.333326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.334792 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.334794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.335900 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:24.335903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.336637 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.336639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:24.336709 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.336713 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:24.336743 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 13:09:24.336787 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":68}
I20260926 13:09:24.337340 7775 raft_consensus.cc:2781] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:24.337402 7775 raft_consensus.cc:482] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:24.337422 7775 raft_consensus.cc:3037] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:24.337421 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fdab9092be4c56ae301e105bd8cb21" candidate_uuid: "2caca43a229c408aa688a866f07250b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb70baa07584f89868a7f334975c060" is_pre_election: true
I20260926 13:09:24.337489 7555 raft_consensus.cc:2454] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2caca43a229c408aa688a866f07250b6 in term 0.
I20260926 13:09:24.338083 7777 raft_consensus.cc:504] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } } peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } }
I20260926 13:09:24.338156 7775 raft_consensus.cc:504] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } } peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } }
I20260926 13:09:24.338224 7777 leader_election.cc:288] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [CANDIDATE]: Term 1 election: Requested vote from peers 7bb70baa07584f89868a7f334975c060 (127.0.98.2:44653), 2caca43a229c408aa688a866f07250b6 (127.0.98.1:40611)
I20260926 13:09:24.338270 7775 leader_election.cc:288] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [CANDIDATE]: Term 1 election: Requested vote from peers c8137509b54b41d78e847082f20b7fb5 (127.0.98.3:35143), 7bb70baa07584f89868a7f334975c060 (127.0.98.2:44653)
I20260926 13:09:24.338348 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fdab9092be4c56ae301e105bd8cb21" candidate_uuid: "c8137509b54b41d78e847082f20b7fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb70baa07584f89868a7f334975c060"
I20260926 13:09:24.338425 7555 raft_consensus.cc:3037] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:24.338491 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fdab9092be4c56ae301e105bd8cb21" candidate_uuid: "c8137509b54b41d78e847082f20b7fb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2caca43a229c408aa688a866f07250b6"
I20260926 13:09:24.338581 7419 raft_consensus.cc:2379] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c8137509b54b41d78e847082f20b7fb5 in current term 1: Already voted for candidate 2caca43a229c408aa688a866f07250b6 in this term.
I20260926 13:09:24.339308 7555 raft_consensus.cc:2454] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8137509b54b41d78e847082f20b7fb5 in term 1.
I20260926 13:09:24.339417 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fdab9092be4c56ae301e105bd8cb21" candidate_uuid: "2caca43a229c408aa688a866f07250b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8137509b54b41d78e847082f20b7fb5"
I20260926 13:09:24.339493 7624 leader_election.cc:302] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bb70baa07584f89868a7f334975c060, c8137509b54b41d78e847082f20b7fb5; no voters: 2caca43a229c408aa688a866f07250b6
I20260926 13:09:24.339504 7691 raft_consensus.cc:2379] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2caca43a229c408aa688a866f07250b6 in current term 1: Already voted for candidate c8137509b54b41d78e847082f20b7fb5 in this term.
I20260926 13:09:24.339618 7777 raft_consensus.cc:2781] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:24.339648 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2fdab9092be4c56ae301e105bd8cb21" candidate_uuid: "2caca43a229c408aa688a866f07250b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb70baa07584f89868a7f334975c060"
I20260926 13:09:24.339733 7554 raft_consensus.cc:2379] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2caca43a229c408aa688a866f07250b6 in current term 1: Already voted for candidate c8137509b54b41d78e847082f20b7fb5 in this term.
I20260926 13:09:24.339807 7777 raft_consensus.cc:686] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 1 LEADER]: Becoming Leader. State: Replica: c8137509b54b41d78e847082f20b7fb5, State: Running, Role: LEADER
I20260926 13:09:24.339972 7352 leader_election.cc:302] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2caca43a229c408aa688a866f07250b6; no voters: 7bb70baa07584f89868a7f334975c060, c8137509b54b41d78e847082f20b7fb5
I20260926 13:09:24.339962 7777 consensus_queue.cc:237] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } } peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } }
I20260926 13:09:24.340061 7775 raft_consensus.cc:2726] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 13:09:24.340813 6499 catalog_manager.cc:5495] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8137509b54b41d78e847082f20b7fb5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c8137509b54b41d78e847082f20b7fb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } health_report { overall_health: UNKNOWN } } }
W20260926 13:09:24.357950 7468 tablet.cc:2369] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:09:24.377094 7740 tablet.cc:2369] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:09:24.382258 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.380073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.0.98.3:35143)
0926 13:09:24.380120 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.380121 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.380128 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:09:24.380163 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.380163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.380182 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.380245 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:24.380345 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.380880 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.381593 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.381772 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.381826 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.381962 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:24.382059 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:24.382125 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.382164 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.382188 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:09:24.384006 7554 raft_consensus.cc:1261] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060 [term 1 FOLLOWER]: Refusing update from remote peer c8137509b54b41d78e847082f20b7fb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:24.384032 7419 raft_consensus.cc:1261] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 1 FOLLOWER]: Refusing update from remote peer c8137509b54b41d78e847082f20b7fb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:24.384315 7780 consensus_queue.cc:1035] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:24.384498 7777 consensus_queue.cc:1035] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:24.387274 7783 mvcc.cc:205] Tried to move back new op lower bound from 7333593761314201600 to 7333593761137074176. Current Snapshot: MvccSnapshot[applied={T|T < 7333593761314201600}]
I20260926 13:09:24.387346 7785 mvcc.cc:205] Tried to move back new op lower bound from 7333593761314201600 to 7333593761137074176. Current Snapshot: MvccSnapshot[applied={T|T < 7333593761314201600}]
I20260926 13:09:24.391292 6499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:24.391412 6499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:24.392938 6499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f40517c7135f4dd6b4f86d931669cc3f: Sending DeleteTablet for 3 replicas of tablet b2fdab9092be4c56ae301e105bd8cb21
I20260926 13:09:24.393313 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet b2fdab9092be4c56ae301e105bd8cb21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57204
I20260926 13:09:24.393285 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet b2fdab9092be4c56ae301e105bd8cb21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51742
I20260926 13:09:24.393569 7794 tablet_replica.cc:330] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5: stopping tablet replica
I20260926 13:09:24.393568 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet b2fdab9092be4c56ae301e105bd8cb21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49230
I20260926 13:09:24.393667 7794 raft_consensus.cc:2229] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:09:24.393781 7794 raft_consensus.cc:2258] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:24.393950 7795 tablet_replica.cc:330] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060: stopping tablet replica
I20260926 13:09:24.394065 7795 raft_consensus.cc:2229] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:24.394173 7795 raft_consensus.cc:2258] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:24.394690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 6468
I20260926 13:09:24.394685 7795 ts_tablet_manager.cc:1905] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:24.396968 7795 ts_tablet_manager.cc:1918] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:24.397073 7795 log.cc:1196] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/b2fdab9092be4c56ae301e105bd8cb21
I20260926 13:09:24.397365 7795 ts_tablet_manager.cc:1939] T b2fdab9092be4c56ae301e105bd8cb21 P 7bb70baa07584f89868a7f334975c060: Deleting consensus metadata
I20260926 13:09:24.399242 7794 ts_tablet_manager.cc:1905] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:09:24.399919 7490 connection.cc:535] server connection from 127.0.0.1:51742 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:09:24.401008 7794 ts_tablet_manager.cc:1918] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:24.401105 7794 log.cc:1196] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/b2fdab9092be4c56ae301e105bd8cb21
I20260926 13:09:24.401373 7794 ts_tablet_manager.cc:1939] T b2fdab9092be4c56ae301e105bd8cb21 P c8137509b54b41d78e847082f20b7fb5: Deleting consensus metadata
W20260926 13:09:24.401782 7626 connection.cc:441] server connection from 127.0.0.1:57204 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57204 (request call id 1) could send its response
I20260926 13:09:24.401821 7796 tablet_replica.cc:330] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6: stopping tablet replica
I20260926 13:09:24.401939 7796 raft_consensus.cc:2229] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:24.402050 7796 raft_consensus.cc:2258] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:24.402832 7796 ts_tablet_manager.cc:1905] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:24.404512 7796 ts_tablet_manager.cc:1918] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:24.404600 7796 log.cc:1196] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-0/wal/wals/b2fdab9092be4c56ae301e105bd8cb21
I20260926 13:09:24.404856 7796 ts_tablet_manager.cc:1939] T b2fdab9092be4c56ae301e105bd8cb21 P 2caca43a229c408aa688a866f07250b6: Deleting consensus metadata
W20260926 13:09:24.405227 7354 connection.cc:441] server connection from 127.0.0.1:49230 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49230 (request call id 1) could send its response
I20260926 13:09:24.409817 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.402146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.0.98.60:35631)
0926 13:09:24.402226 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.402227 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.402236 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:24.402263 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.402264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.402277 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.402326 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:24.403226 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.404008 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.405131 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.405439 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.405481 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.407271 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:24.407273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:24.407273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:24.407280 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:24.408640 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.408967 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.408968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.408968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.409053 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.409251 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.409253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.409253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.409391 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:24.409465 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.409655 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.409748 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 13:09:24.410502 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:24.402188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.0.98.61:45555)
0926 13:09:24.402380 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:24.402382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:24.402392 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:24.402416 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:24.402417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:24.402429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:24.402473 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:24.403225 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.404003 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.405195 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:24.405504 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:24.405542 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:24.407265 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:24.407267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:24.407267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:24.407276 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:24.408888 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.409456 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.409457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.409457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.409520 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:24.409700 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:24.409701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:24.409701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:24.409832 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:24.409924 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:24.410325 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 13:09:24.410415 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260926 13:09:25.388778 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.388368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.2:37097)
0926 13:09:25.388551 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.388700 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
W20260926 13:09:25.389110 7602 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45001 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
I20260926 13:09:25.389122 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.388755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.1:58887)
0926 13:09:25.388878 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.389063 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":52,"threads_started":1}
W20260926 13:09:25.389367 7466 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45001 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
I20260926 13:09:25.389971 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.389629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.3:52675)
0926 13:09:25.389828 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.389918 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
W20260926 13:09:25.390261 7738 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45001 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
I20260926 13:09:25.893289 7807 raft_consensus.cc:482] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f40517c7135f4dd6b4f86d931669cc3f)
I20260926 13:09:25.893363 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f40517c7135f4dd6b4f86d931669cc3f)
I20260926 13:09:25.893438 7807 raft_consensus.cc:504] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } }
I20260926 13:09:25.893499 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } }
I20260926 13:09:25.893680 7807 leader_election.cc:288] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f40517c7135f4dd6b4f86d931669cc3f (127.0.98.62:45001), dea95093e5a546bda5ade0b7aa86c648 (127.0.98.60:35631)
I20260926 13:09:25.893827 7806 leader_election.cc:288] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f40517c7135f4dd6b4f86d931669cc3f (127.0.98.62:45001), c54404f73c1e40229892ba9529e9a003 (127.0.98.61:45555)
I20260926 13:09:25.894183 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.893862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33548)
0926 13:09:25.894054 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.894129 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":63,"threads_started":1}
I20260926 13:09:25.894227 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.893963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33556)
0926 13:09:25.894144 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.894184 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
W20260926 13:09:25.894398 7268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
W20260926 13:09:25.894474 7081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
I20260926 13:09:25.895102 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.894982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33558)
0926 13:09:25.895039 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.895066 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 13:09:25.895288 7268 leader_election.cc:334] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f40517c7135f4dd6b4f86d931669cc3f (127.0.98.62:45001): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
I20260926 13:09:25.895368 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.895203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33562)
0926 13:09:25.895292 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.895323 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260926 13:09:25.895494 7081 leader_election.cc:334] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f40517c7135f4dd6b4f86d931669cc3f (127.0.98.62:45001): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
I20260926 13:09:25.897511 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.894035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45555 (local address 127.0.0.1:45802)
0926 13:09:25.894169 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.894179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:25.894244 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:25.894701 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:25.894702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:25.894706 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:25.894774 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:25.894777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.895771 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:25.895772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:25.896614 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:25.896616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.897260 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:25.897261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:25.897325 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:25.897329 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:25.897374 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 13:09:25.897436 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":38,"threads_started":1}
I20260926 13:09:25.897521 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.894137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.0.98.61:45555)
0926 13:09:25.894465 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:25.894467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:25.894477 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:25.894512 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:25.894513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:25.894526 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:25.894868 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:25.894924 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.895683 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:25.896713 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.897181 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:25.897247 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:25.897261 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:25.897393 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 13:09:25.897436 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":40,"threads_started":1}
I20260926 13:09:25.897755 7107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dea95093e5a546bda5ade0b7aa86c648" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c54404f73c1e40229892ba9529e9a003" is_pre_election: true
I20260926 13:09:25.897868 7107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea95093e5a546bda5ade0b7aa86c648 in term 1.
I20260926 13:09:25.897994 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.894047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35631 (local address 127.0.0.1:43060)
0926 13:09:25.894369 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.894381 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:25.894449 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:25.894741 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:25.894743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:25.894745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:25.894817 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:25.894819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.895924 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:25.895925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:25.897053 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:25.897055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.897675 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:25.897676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:25.897744 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:25.897751 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:09:25.897779 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 13:09:25.897848 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":65,"threads_started":1}
I20260926 13:09:25.898064 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.894344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.0.98.60:35631)
0926 13:09:25.894575 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:25.894578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:25.894586 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:25.894618 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:25.894620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:25.894632 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:25.894730 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:25.894955 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.895776 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:25.897139 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.897570 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:25.897910 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:25.897928 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:25.897945 (+ 17us) server_negotiation.cc:299] Negotiation successful
0926 13:09:25.897995 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":209,"thread_start_us":42,"threads_started":1}
I20260926 13:09:25.898089 7269 leader_election.cc:302] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c54404f73c1e40229892ba9529e9a003, dea95093e5a546bda5ade0b7aa86c648; no voters: f40517c7135f4dd6b4f86d931669cc3f
I20260926 13:09:25.898319 7806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 13:09:25.898358 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f40517c7135f4dd6b4f86d931669cc3f)
I20260926 13:09:25.898379 7806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:09:25.898470 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c54404f73c1e40229892ba9529e9a003" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dea95093e5a546bda5ade0b7aa86c648" is_pre_election: true
I20260926 13:09:25.928817 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } }
I20260926 13:09:25.929055 7294 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c54404f73c1e40229892ba9529e9a003 in current term 2: Already voted for candidate dea95093e5a546bda5ade0b7aa86c648 in this term.
I20260926 13:09:25.929097 7806 leader_election.cc:288] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [CANDIDATE]: Term 2 election: Requested vote from peers f40517c7135f4dd6b4f86d931669cc3f (127.0.98.62:45001), c54404f73c1e40229892ba9529e9a003 (127.0.98.61:45555)
I20260926 13:09:25.929713 7081 leader_election.cc:302] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c54404f73c1e40229892ba9529e9a003; no voters: dea95093e5a546bda5ade0b7aa86c648, f40517c7135f4dd6b4f86d931669cc3f
I20260926 13:09:25.929785 7107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dea95093e5a546bda5ade0b7aa86c648" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c54404f73c1e40229892ba9529e9a003"
I20260926 13:09:25.929875 7807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:09:25.930060 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.929461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33578)
0926 13:09:25.929975 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.930019 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486}
I20260926 13:09:25.930537 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.930394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33582)
0926 13:09:25.930473 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.930515 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 13:09:25.930614 7268 leader_election.cc:334] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f40517c7135f4dd6b4f86d931669cc3f (127.0.98.62:45001): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
I20260926 13:09:25.930989 7807 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:09:25.961524 7107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea95093e5a546bda5ade0b7aa86c648 in term 2.
I20260926 13:09:25.962316 7269 leader_election.cc:302] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c54404f73c1e40229892ba9529e9a003, dea95093e5a546bda5ade0b7aa86c648; no voters: f40517c7135f4dd6b4f86d931669cc3f
I20260926 13:09:25.962502 7806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:09:25.962738 7806 raft_consensus.cc:686] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [term 2 LEADER]: Becoming Leader. State: Replica: dea95093e5a546bda5ade0b7aa86c648, State: Running, Role: LEADER
I20260926 13:09:25.962828 7806 consensus_queue.cc:237] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } }
I20260926 13:09:25.963369 7816 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dea95093e5a546bda5ade0b7aa86c648. Latest consensus state: current_term: 2 leader_uuid: "dea95093e5a546bda5ade0b7aa86c648" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } } }
I20260926 13:09:25.963528 7816 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:25.964545 7818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:25.964999 7818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:25.965068 7818 catalog_manager.cc:1259] Loaded cluster ID: 412444a8b1f04915b5763a001a60229f
I20260926 13:09:25.965095 7818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:25.965248 7818 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:25.965308 7818 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: Loaded TSK: 0
I20260926 13:09:25.965399 7818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:09:25.989467 7284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:25.991811 7284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:25.993304 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.993115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33602)
0926 13:09:25.993213 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.993264 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 13:09:25.993275 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [term 2 FOLLOWER]: Refusing update from remote peer dea95093e5a546bda5ade0b7aa86c648: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 13:09:25.993633 7816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 13:09:25.993717 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.993610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.0.1:33604)
0926 13:09:25.993659 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.993689 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 13:09:25.993821 7268 consensus_peers.cc:487] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 -> Peer f40517c7135f4dd6b4f86d931669cc3f (127.0.98.62:45001): Couldn't send request to peer f40517c7135f4dd6b4f86d931669cc3f. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 13:09:25.994405 7807 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dea95093e5a546bda5ade0b7aa86c648. Latest consensus state: current_term: 2 leader_uuid: "dea95093e5a546bda5ade0b7aa86c648" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } } }
I20260926 13:09:25.994527 7807 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:09:25.995026 7806 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "dea95093e5a546bda5ade0b7aa86c648" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } } }
I20260926 13:09:25.995136 7806 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:25.995290 7807 sys_catalog.cc:455] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "dea95093e5a546bda5ade0b7aa86c648" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f40517c7135f4dd6b4f86d931669cc3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45001 } } peers { permanent_uuid: "c54404f73c1e40229892ba9529e9a003" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45555 } attrs { promote: false } } peers { permanent_uuid: "dea95093e5a546bda5ade0b7aa86c648" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35631 } attrs { promote: false } } }
I20260926 13:09:25.995384 7807 sys_catalog.cc:458] T 00000000000000000000000000000000 P c54404f73c1e40229892ba9529e9a003 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:09:26.001291 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.997771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44653 (local address 127.0.0.1:51752)
0926 13:09:25.997829 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.997841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:25.997880 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:25.998358 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:25.998360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:25.998362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:25.998415 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:25.998462 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.999512 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:25.999514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:26.000401 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:26.000404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.001036 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.001037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:26.001105 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:26.001112 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:09:26.001139 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 13:09:26.001204 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 13:09:26.001639 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.997850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.0.98.2:44653)
0926 13:09:25.998204 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:25.998206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:25.998216 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:25.998252 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:25.998253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:25.998265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:25.998339 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:25.998563 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.999402 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.000492 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.000933 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.001477 (+ 544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:26.001494 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:26.001518 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 13:09:26.001567 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":70,"threads_started":1}
I20260926 13:09:26.001850 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.997793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40611 (local address 127.0.0.1:49240)
0926 13:09:25.997903 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.997915 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:25.997974 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:25.998381 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:25.998382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:25.998385 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:25.998415 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:25.998417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.999474 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:25.999475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:26.000772 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:26.000774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.001659 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.001661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:26.001729 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:26.001733 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:26.001761 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 13:09:26.001810 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":61}
I20260926 13:09:26.001961 7535 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab62de317e84e30a61e4e66aaf77e29 (DEFAULT_TABLE table=test-table [id=9a3b103f914f4c148430f43599e2feb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:26.002112 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab62de317e84e30a61e4e66aaf77e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:26.002172 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.998061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.0.98.1:40611)
0926 13:09:25.998235 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:25.998238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:25.998250 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:09:25.998276 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:25.998276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:25.998287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:25.998477 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:25.998544 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.999378 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.000884 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.001540 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.002034 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:26.002050 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:26.002069 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 13:09:26.002107 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260926 13:09:26.002188 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.997802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35143 (local address 127.0.0.1:57222)
0926 13:09:25.997988 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:25.997998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:25.998047 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:25.998429 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:25.998430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:25.998433 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:25.998501 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:25.998504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.999636 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:25.999636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:26.000772 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:26.000774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.001953 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.001955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:26.002026 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:26.002032 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:09:26.002062 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 13:09:26.002113 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260926 13:09:26.002444 7399 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab62de317e84e30a61e4e66aaf77e29 (DEFAULT_TABLE table=test-table [id=9a3b103f914f4c148430f43599e2feb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:26.002507 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:25.998116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.0.98.3:35143)
0926 13:09:25.998300 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:25.998302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:25.998312 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:25.998340 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:25.998341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:25.998352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:25.998411 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:25.998625 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:25.999379 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.000959 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.001689 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.002336 (+ 647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:26.002357 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:26.002378 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 13:09:26.002431 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260926 13:09:26.002607 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab62de317e84e30a61e4e66aaf77e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:26.002698 7671 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab62de317e84e30a61e4e66aaf77e29 (DEFAULT_TABLE table=test-table [id=9a3b103f914f4c148430f43599e2feb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:26.002828 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab62de317e84e30a61e4e66aaf77e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:26.003691 7828 tablet_bootstrap.cc:492] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060: Bootstrap starting.
I20260926 13:09:26.004112 7828 tablet_bootstrap.cc:654] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:26.004451 7829 tablet_bootstrap.cc:492] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6: Bootstrap starting.
I20260926 13:09:26.004631 7831 tablet_bootstrap.cc:492] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5: Bootstrap starting.
I20260926 13:09:26.004691 7828 tablet_bootstrap.cc:492] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060: No bootstrap required, opened a new log
I20260926 13:09:26.004752 7828 ts_tablet_manager.cc:1397] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:26.004909 7828 raft_consensus.cc:348] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } } peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } }
I20260926 13:09:26.004998 7828 raft_consensus.cc:374] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:26.005023 7828 raft_consensus.cc:729] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb70baa07584f89868a7f334975c060, State: Initialized, Role: FOLLOWER
I20260926 13:09:26.005069 7828 consensus_queue.cc:260] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } } peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } }
I20260926 13:09:26.005183 7831 tablet_bootstrap.cc:654] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:26.005230 7828 ts_tablet_manager.cc:1428] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:26.005328 7829 tablet_bootstrap.cc:654] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:26.005975 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.005687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.2:55619)
0926 13:09:26.005862 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.005912 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
I20260926 13:09:26.006134 7831 tablet_bootstrap.cc:492] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5: No bootstrap required, opened a new log
I20260926 13:09:26.006196 7831 ts_tablet_manager.cc:1397] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:26.006328 7831 raft_consensus.cc:348] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } } peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } }
I20260926 13:09:26.006412 7831 raft_consensus.cc:374] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:26.006433 7831 raft_consensus.cc:729] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8137509b54b41d78e847082f20b7fb5, State: Initialized, Role: FOLLOWER
I20260926 13:09:26.006500 7831 consensus_queue.cc:260] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } } peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } }
I20260926 13:09:26.006558 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.006439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.2:56479)
0926 13:09:26.006501 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.006535 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 13:09:26.006654 7829 tablet_bootstrap.cc:492] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6: No bootstrap required, opened a new log
I20260926 13:09:26.006683 7831 ts_tablet_manager.cc:1428] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:26.006712 7829 ts_tablet_manager.cc:1397] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:26.006880 7829 raft_consensus.cc:348] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } } peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } }
I20260926 13:09:26.006963 7829 raft_consensus.cc:374] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:26.006980 7829 raft_consensus.cc:729] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2caca43a229c408aa688a866f07250b6, State: Initialized, Role: FOLLOWER
I20260926 13:09:26.007025 7829 consensus_queue.cc:260] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } } peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } }
I20260926 13:09:26.007190 7829 ts_tablet_manager.cc:1428] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:26.007486 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.006969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.3:49465)
0926 13:09:26.007391 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.007445 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":93,"threads_started":1}
I20260926 13:09:26.007498 7737 heartbeater.cc:503] Master 127.0.98.60:35631 was elected leader, sending a full tablet report...
I20260926 13:09:26.007928 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.007466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.1:50167)
0926 13:09:26.007839 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.007879 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":89,"threads_started":1}
I20260926 13:09:26.008059 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.007770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.3:36727)
0926 13:09:26.008001 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.008032 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260926 13:09:26.009423 7832 raft_consensus.cc:482] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:26.009498 7832 raft_consensus.cc:504] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } } peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } }
I20260926 13:09:26.009780 7832 leader_election.cc:288] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2caca43a229c408aa688a866f07250b6 (127.0.98.1:40611), c8137509b54b41d78e847082f20b7fb5 (127.0.98.3:35143)
I20260926 13:09:26.013228 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.010071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51427 (local address 127.0.98.1:40611)
0926 13:09:26.010105 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:26.010106 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:26.010307 (+ 201us) server_negotiation.cc:407] Connection header received
0926 13:09:26.010339 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:26.010340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:26.010354 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:26.010402 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:26.010506 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.011253 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.012520 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.012901 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.012962 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:26.012976 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:26.013119 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 13:09:26.013157 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:09:26.013239 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.010025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40611 (local address 127.0.98.2:51427)
0926 13:09:26.010211 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.010223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:26.010281 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:26.010414 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:26.010415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:26.010416 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:26.010451 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:26.010451 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.011354 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.011355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:26.012424 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:26.012426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.012982 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.012983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:26.013057 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:26.013061 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:26.013106 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 13:09:26.013159 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":1,"thread_start_us":120,"threads_started":1}
I20260926 13:09:26.013329 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.010007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44439 (local address 127.0.98.3:35143)
0926 13:09:26.010051 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:26.010054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:26.010067 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:09:26.010103 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:26.010104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:26.010119 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:26.010187 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:26.010364 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.011222 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.012530 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.012942 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.013041 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:26.013056 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:26.013209 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 13:09:26.013253 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:09:26.013523 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab62de317e84e30a61e4e66aaf77e29" candidate_uuid: "7bb70baa07584f89868a7f334975c060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2caca43a229c408aa688a866f07250b6" is_pre_election: true
I20260926 13:09:26.013636 7419 raft_consensus.cc:2454] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bb70baa07584f89868a7f334975c060 in term 0.
I20260926 13:09:26.013797 7491 leader_election.cc:302] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2caca43a229c408aa688a866f07250b6, 7bb70baa07584f89868a7f334975c060; no voters:
I20260926 13:09:26.013882 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.009938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35143 (local address 127.0.98.2:44439)
0926 13:09:26.009982 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.009992 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:26.010043 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:26.010211 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:26.010212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:26.010214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:26.010283 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:26.010285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.011316 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.011318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:26.012436 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:26.012441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.013057 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.013058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:26.013147 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:26.013150 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:26.013194 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 13:09:26.013253 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":72}
I20260926 13:09:26.013967 7832 raft_consensus.cc:2781] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:26.014014 7832 raft_consensus.cc:482] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:26.014038 7832 raft_consensus.cc:3037] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:26.014109 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab62de317e84e30a61e4e66aaf77e29" candidate_uuid: "7bb70baa07584f89868a7f334975c060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8137509b54b41d78e847082f20b7fb5" is_pre_election: true
I20260926 13:09:26.014209 7691 raft_consensus.cc:2454] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bb70baa07584f89868a7f334975c060 in term 0.
I20260926 13:09:26.028095 7832 raft_consensus.cc:504] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } } peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } }
I20260926 13:09:26.028540 7832 leader_election.cc:288] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [CANDIDATE]: Term 1 election: Requested vote from peers 2caca43a229c408aa688a866f07250b6 (127.0.98.1:40611), c8137509b54b41d78e847082f20b7fb5 (127.0.98.3:35143)
I20260926 13:09:26.028577 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab62de317e84e30a61e4e66aaf77e29" candidate_uuid: "7bb70baa07584f89868a7f334975c060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2caca43a229c408aa688a866f07250b6"
I20260926 13:09:26.028744 7419 raft_consensus.cc:3037] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:26.028843 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab62de317e84e30a61e4e66aaf77e29" candidate_uuid: "7bb70baa07584f89868a7f334975c060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8137509b54b41d78e847082f20b7fb5"
I20260926 13:09:26.028997 7691 raft_consensus.cc:3037] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:26.029637 7419 raft_consensus.cc:2454] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb70baa07584f89868a7f334975c060 in term 1.
I20260926 13:09:26.029842 7491 leader_election.cc:302] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2caca43a229c408aa688a866f07250b6, 7bb70baa07584f89868a7f334975c060; no voters:
I20260926 13:09:26.029970 7832 raft_consensus.cc:2781] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:26.030081 7691 raft_consensus.cc:2454] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bb70baa07584f89868a7f334975c060 in term 1.
I20260926 13:09:26.030179 7832 raft_consensus.cc:686] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 1 LEADER]: Becoming Leader. State: Replica: 7bb70baa07584f89868a7f334975c060, State: Running, Role: LEADER
I20260926 13:09:26.030314 7601 heartbeater.cc:503] Master 127.0.98.60:35631 was elected leader, sending a full tablet report...
I20260926 13:09:26.030279 7832 consensus_queue.cc:237] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } } peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } }
I20260926 13:09:26.031080 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.030766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.2:58289)
0926 13:09:26.030864 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.031015 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 13:09:26.031651 7283 catalog_manager.cc:5495] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bb70baa07584f89868a7f334975c060 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7bb70baa07584f89868a7f334975c060" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb70baa07584f89868a7f334975c060" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 } health_report { overall_health: UNKNOWN } } }
I20260926 13:09:26.041565 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.039235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51754 (local address 127.0.98.2:44653)
0926 13:09:26.039295 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:26.039297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:26.039307 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:26.039360 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:26.039361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:26.039389 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:26.039447 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:26.039559 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.040065 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.040789 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:26.040986 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:26.041040 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:26.041185 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:26.041296 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:26.041400 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:26.041417 (+ 17us) server_negotiation.cc:299] Negotiation successful
0926 13:09:26.041447 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:09:26.042990 7691 raft_consensus.cc:1261] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5 [term 1 FOLLOWER]: Refusing update from remote peer 7bb70baa07584f89868a7f334975c060: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:26.043025 7419 raft_consensus.cc:1261] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6 [term 1 FOLLOWER]: Refusing update from remote peer 7bb70baa07584f89868a7f334975c060: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:26.043413 7841 consensus_queue.cc:1035] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2caca43a229c408aa688a866f07250b6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:26.043578 7832 consensus_queue.cc:1035] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8137509b54b41d78e847082f20b7fb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:26.043988 7465 heartbeater.cc:503] Master 127.0.98.60:35631 was elected leader, sending a full tablet report...
I20260926 13:09:26.044669 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.044520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.1:53287)
0926 13:09:26.044581 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.044625 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 13:09:26.045305 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.045161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.3:54467)
0926 13:09:26.045210 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.045256 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 13:09:26.045562 7845 mvcc.cc:205] Tried to move back new op lower bound from 7333593768109219840 to 7333593768060948480. Current Snapshot: MvccSnapshot[applied={T|T < 7333593768109219840}]
I20260926 13:09:26.046020 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.045905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.2:59585)
0926 13:09:26.045954 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.045983 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 13:09:26.046486 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.046369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.1:42511)
0926 13:09:26.046415 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.046456 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 13:09:26.048043 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.047940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.2:59511)
0926 13:09:26.047984 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.048010 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 13:09:26.048082 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.047935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45001 (local address 127.0.98.3:59071)
0926 13:09:26.048017 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.048045 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 13:09:26.048617 7283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:26.048714 7283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:26.050040 7283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dea95093e5a546bda5ade0b7aa86c648: Sending DeleteTablet for 3 replicas of tablet 3ab62de317e84e30a61e4e66aaf77e29
I20260926 13:09:26.050376 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab62de317e84e30a61e4e66aaf77e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:26 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:51752
I20260926 13:09:26.050591 7854 tablet_replica.cc:330] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060: stopping tablet replica
I20260926 13:09:26.050652 7854 raft_consensus.cc:2229] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:09:26.050761 7854 raft_consensus.cc:2258] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:26.050747 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab62de317e84e30a61e4e66aaf77e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:26 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49240
I20260926 13:09:26.050993 7855 tablet_replica.cc:330] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6: stopping tablet replica
I20260926 13:09:26.051093 7854 ts_tablet_manager.cc:1905] T 3ab62de317e84e30a61e4e66aaf77e29 P 7bb70baa07584f89868a7f334975c060: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:26.051110 7855 raft_consensus.cc:2229] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:26.051237 7855 raft_consensus.cc:2258] T 3ab62de317e84e30a61e4e66aaf77e29 P 2caca43a229c408aa688a866f07250b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:26.051434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 7336
I20260926 13:09:26.054404 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab62de317e84e30a61e4e66aaf77e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:26 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57222
I20260926 13:09:26.054695 7856 tablet_replica.cc:330] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5: stopping tablet replica
I20260926 13:09:26.054785 7856 raft_consensus.cc:2229] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:26.054872 7856 raft_consensus.cc:2258] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:26.055253 7856 ts_tablet_manager.cc:1905] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:26.057955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 7472
I20260926 13:09:26.058051 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.057876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40611 (local address 127.0.0.1:49242)
0926 13:09:26.057961 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.058000 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 13:09:26.059206 7271 catalog_manager.cc:4584] TS 2caca43a229c408aa688a866f07250b6 (127.0.98.1:40611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ab62de317e84e30a61e4e66aaf77e29: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40611: connect: Connection refused (error 111)
I20260926 13:09:26.112035 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.111838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40611 (local address 127.0.0.1:49254)
0926 13:09:26.111934 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.111995 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 13:09:26.112529 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.112444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40611 (local address 127.0.0.1:49258)
0926 13:09:26.112487 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.112514 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 13:09:26.131055 7856 ts_tablet_manager.cc:1918] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:26.131155 7856 log.cc:1196] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/3ab62de317e84e30a61e4e66aaf77e29
I20260926 13:09:26.131474 7856 ts_tablet_manager.cc:1939] T 3ab62de317e84e30a61e4e66aaf77e29 P c8137509b54b41d78e847082f20b7fb5: Deleting consensus metadata
I20260926 13:09:26.131508 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.131334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44653 (local address 127.0.0.1:51758)
0926 13:09:26.131416 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.131461 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 13:09:26.131474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 7608
I20260926 13:09:26.139739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 6964
I20260926 13:09:26.139909 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:26.139718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35143 (local address 127.0.0.1:57238)
0926 13:09:26.139788 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:26.139838 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:35143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 13:09:26.144893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 7151
2026-09-26T13:09:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4770 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:26 dist-test-slave-2dw5 krb5kdc[7859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:26 dist-test-slave-2dw5 krb5kdc[7859](info): set up 2 sockets
Sep 26 13:09:26 dist-test-slave-2dw5 krb5kdc[7859](info): commencing operation
krb5kdc: starting...
W20260926 13:09:27.607251 392 mini_kdc.cc:121] Time spent starting KDC: real 1.442s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:27 dist-test-slave-2dw5 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 13:09:27.727131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38579
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:27.807513 7872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:27.807754 7872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:27.807788 7872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:27.809053 7872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:27.809104 7872 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 13:09:27.809120 7872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:27.809139 7872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:27.809175 7872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:27.809194 7872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:27.810801 7872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38579
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:27.811092 7872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:27.811305 7872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:27.816606 7880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:27.816628 7877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:27.816692 7878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:27.816869 7872 server_base.cc:1034] running on GCE node
I20260926 13:09:27.817124 7872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:27.817513 7872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:27.818704 7872 hybrid_clock.cc:648] HybridClock initialized: now 1790428167818685 us; error 32 us; skew 500 ppm
Sep 26 13:09:27 dist-test-slave-2dw5 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:27.821663 7872 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 13:09:27.822089 7872 webserver.cc:460] Webserver started at http://127.0.98.62:37353/ using document root <none> and password file <none>
I20260926 13:09:27.822273 7872 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:27.822319 7872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:27.822417 7872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:27.823696 7872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "cb26d5dde41a4ff59bdf0101372a5abc"
format_stamp: "Formatted at 2026-09-26 13:09:27 on dist-test-slave-2dw5"
server_key: "612633bd6acfd4ffcc9fe65a3aaeea86"
server_key_iv: "7f4a220228ef7c7c39de26e843dba26a"
server_key_version: "encryptionkey@0"
I20260926 13:09:27.824126 7872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "cb26d5dde41a4ff59bdf0101372a5abc"
format_stamp: "Formatted at 2026-09-26 13:09:27 on dist-test-slave-2dw5"
server_key: "612633bd6acfd4ffcc9fe65a3aaeea86"
server_key_iv: "7f4a220228ef7c7c39de26e843dba26a"
server_key_version: "encryptionkey@0"
I20260926 13:09:27.825511 7872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:09:27.826139 7887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:27.826311 7872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:27.826398 7872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "cb26d5dde41a4ff59bdf0101372a5abc"
format_stamp: "Formatted at 2026-09-26 13:09:27 on dist-test-slave-2dw5"
server_key: "612633bd6acfd4ffcc9fe65a3aaeea86"
server_key_iv: "7f4a220228ef7c7c39de26e843dba26a"
server_key_version: "encryptionkey@0"
I20260926 13:09:27.826473 7872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:27.845153 7872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:27.847258 7872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:27.847379 7872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:27.851840 7872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38579
I20260926 13:09:27.851935 7949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38579 every 8 connection(s)
I20260926 13:09:27.852231 7872 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:27.852898 7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:27.855266 7950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc: Bootstrap starting.
I20260926 13:09:27.855813 7950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:27.856055 7950 log.cc:824] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:27.856767 7950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc: No bootstrap required, opened a new log
I20260926 13:09:27.859067 7950 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb26d5dde41a4ff59bdf0101372a5abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38579 } }
I20260926 13:09:27.859233 7950 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:27.859256 7950 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb26d5dde41a4ff59bdf0101372a5abc, State: Initialized, Role: FOLLOWER
I20260926 13:09:27.859366 7950 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb26d5dde41a4ff59bdf0101372a5abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38579 } }
I20260926 13:09:27.859422 7950 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:27.859450 7950 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:27.859489 7950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:27.860091 7950 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb26d5dde41a4ff59bdf0101372a5abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38579 } }
I20260926 13:09:27.860191 7950 leader_election.cc:302] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb26d5dde41a4ff59bdf0101372a5abc; no voters:
I20260926 13:09:27.860378 7950 leader_election.cc:288] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:27.860469 7953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:27.860634 7950 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:27.860639 7953 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [term 1 LEADER]: Becoming Leader. State: Replica: cb26d5dde41a4ff59bdf0101372a5abc, State: Running, Role: LEADER
I20260926 13:09:27.860759 7953 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb26d5dde41a4ff59bdf0101372a5abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38579 } }
I20260926 13:09:27.861130 7954 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb26d5dde41a4ff59bdf0101372a5abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb26d5dde41a4ff59bdf0101372a5abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38579 } } }
I20260926 13:09:27.861212 7954 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:27.861395 7962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:27.861433 7955 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb26d5dde41a4ff59bdf0101372a5abc. Latest consensus state: current_term: 1 leader_uuid: "cb26d5dde41a4ff59bdf0101372a5abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb26d5dde41a4ff59bdf0101372a5abc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38579 } } }
I20260926 13:09:27.861822 7955 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:27.861929 7962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:27.861927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 7872
I20260926 13:09:27.862010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:27.862224 392 external_mini_cluster.cc:1428] Setting key 4b0c199740e5fed5e6b5cc701084c0ac
I20260926 13:09:27.863857 7962 catalog_manager.cc:1347] Generated new cluster ID: 84f702d1e5df4dcf8648d5735ea24d80
I20260926 13:09:27.863915 7962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:09:27 dist-test-slave-2dw5 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:27.871668 7962 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:27.872349 7962 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:27.875588 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:27.862911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.0.98.62:38579)
0926 13:09:27.863142 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:27.863152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:27.863203 (+ 51us) server_negotiation.cc:407] Connection header received
0926 13:09:27.863341 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:27.863350 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:27.863409 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:27.863563 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:27.864346 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:27.865874 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:27.867122 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:27.867831 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:27.867952 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:27.871899 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:27.871904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:27.871904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:27.871918 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:27.873910 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:27.874336 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:27.874340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:27.874340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:27.874458 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:27.874640 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:27.874641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:27.874641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:27.874936 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:27.875056 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:27.875184 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 13:09:27.875345 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260926 13:09:27.883327 7962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc: Generated new TSK 0
I20260926 13:09:27.883540 7962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 13:09:27.913388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38579
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:27.996074 7976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:27.996315 7976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:27.996342 7976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:27.997627 7976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:27.997677 7976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:27.997705 7976 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:09:27.997725 7976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:27.999387 7976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:27.999650 7976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:27.999890 7976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:28.005144 7982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:28.005376 7981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:28.005249 7984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:28.005457 7976 server_base.cc:1034] running on GCE node
I20260926 13:09:28.005649 7976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:28.005891 7976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:28.007064 7976 hybrid_clock.cc:648] HybridClock initialized: now 1790428168007043 us; error 28 us; skew 500 ppm
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:28.009737 7976 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 13:09:28.010154 7976 webserver.cc:460] Webserver started at http://127.0.98.1:39469/ using document root <none> and password file <none>
I20260926 13:09:28.010360 7976 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:28.010401 7976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:28.010493 7976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:28.011560 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "427df8ff38bf403ca17e899807721af3"
format_stamp: "Formatted at 2026-09-26 13:09:28 on dist-test-slave-2dw5"
server_key: "1f5e9cfbcbc878bdec5827a6ff3c6b83"
server_key_iv: "a284ecd2b4983d49f2f8add8bb5a9164"
server_key_version: "encryptionkey@0"
I20260926 13:09:28.011840 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "427df8ff38bf403ca17e899807721af3"
format_stamp: "Formatted at 2026-09-26 13:09:28 on dist-test-slave-2dw5"
server_key: "1f5e9cfbcbc878bdec5827a6ff3c6b83"
server_key_iv: "a284ecd2b4983d49f2f8add8bb5a9164"
server_key_version: "encryptionkey@0"
I20260926 13:09:28.013064 7976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:28.013913 7991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.014092 7976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:28.014168 7976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "427df8ff38bf403ca17e899807721af3"
format_stamp: "Formatted at 2026-09-26 13:09:28 on dist-test-slave-2dw5"
server_key: "1f5e9cfbcbc878bdec5827a6ff3c6b83"
server_key_iv: "a284ecd2b4983d49f2f8add8bb5a9164"
server_key_version: "encryptionkey@0"
I20260926 13:09:28.014225 7976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:28.032516 7976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:28.034731 7976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:28.034888 7976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:28.035367 7976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:28.035404 7976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.035436 7976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:28.035460 7976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.042038 7976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32795
I20260926 13:09:28.042116 8106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32795 every 8 connection(s)
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:28.042435 7976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:09:28.045656 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.035912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56005 (local address 127.0.98.62:38579)
0926 13:09:28.035978 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.035980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.036173 (+ 193us) server_negotiation.cc:407] Connection header received
0926 13:09:28.036840 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.036842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.036861 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.036893 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:28.037878 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.038660 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.039776 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.040091 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.040157 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.042970 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:28.042971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:28.042971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:28.042979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:28.044203 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.044601 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.044601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.044602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.044653 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.044936 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.044936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.044937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.045077 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:28.045158 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.045511 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.045591 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:09:28.045869 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.035879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38579 (local address 127.0.98.1:56005)
0926 13:09:28.036043 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.036084 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.036668 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.037098 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.037102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.037450 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:28.037754 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.037762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.038763 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.038765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.039681 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.039683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.040191 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.040192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.040364 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.040994 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:28.041006 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:28.042878 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:28.044288 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.044295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.044297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.044532 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.044743 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.044744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.044744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.044859 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.045172 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:28.045177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:28.045359 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.045492 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.045630 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20260926 13:09:28.046348 8107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38579
I20260926 13:09:28.046435 8107 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:28.046620 8107 heartbeater.cc:511] Master 127.0.98.62:38579 requested a full tablet report, sending...
I20260926 13:09:28.047187 7903 ts_manager.cc:194] Registered new tserver with Master: 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795)
I20260926 13:09:28.047961 7903 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56005
I20260926 13:09:28.048817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 7976
I20260926 13:09:28.048914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:28.049098 392 external_mini_cluster.cc:1428] Setting key 3574b6d1e1e25297c6720d8cd51641a9
I20260926 13:09:28.052837 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.047380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38579 (local address 127.0.98.1:44181)
0926 13:09:28.047488 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.047498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.047555 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.047664 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.047664 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.047815 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:28.047864 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.047865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.048635 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.048637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.049292 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.049295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.049649 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.049650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.049715 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.050146 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:28.050151 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:28.050439 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:28.051825 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.051826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.051826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.051992 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.052261 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.052262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.052263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.052323 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.052575 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:28.052575 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:28.052614 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.052668 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.052759 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":62,"threads_started":1}
I20260926 13:09:28.052852 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.047446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44181 (local address 127.0.98.62:38579)
0926 13:09:28.047483 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.047484 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.047540 (+ 56us) server_negotiation.cc:407] Connection header received
0926 13:09:28.047591 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.047592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.047601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.047661 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:28.047915 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.048542 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.049361 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.049571 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.049624 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.050523 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:28.050524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:28.050524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:28.050530 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:28.051751 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.052050 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.052051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.052051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.052113 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.052384 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.052385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.052385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.052510 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:28.052569 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.052698 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.052772 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 13:09:28.074242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38579
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:28.154544 8116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:28.154755 8116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:28.154779 8116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:28.155961 8116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:28.156004 8116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:28.156025 8116 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:09:28.156042 8116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:28.157502 8116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:28.157691 8116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:28.157874 8116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:28.162734 8124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:28.162734 8121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:28.162912 8116 server_base.cc:1034] running on GCE node
W20260926 13:09:28.162765 8122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:28.163185 8116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:28.163419 8116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:28.164570 8116 hybrid_clock.cc:648] HybridClock initialized: now 1790428168164557 us; error 30 us; skew 500 ppm
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:28.167037 8116 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 13:09:28.167448 8116 webserver.cc:460] Webserver started at http://127.0.98.2:33017/ using document root <none> and password file <none>
I20260926 13:09:28.167665 8116 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:28.167719 8116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:28.167842 8116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:28.168905 8116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "ce3025b2247d423197b85ea0d89d5447"
format_stamp: "Formatted at 2026-09-26 13:09:28 on dist-test-slave-2dw5"
server_key: "e3939d3baf057195bbf5f4d507036dc7"
server_key_iv: "6f222b5c34b81e78f2c6e7a5ab4cca21"
server_key_version: "encryptionkey@0"
I20260926 13:09:28.169257 8116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ce3025b2247d423197b85ea0d89d5447"
format_stamp: "Formatted at 2026-09-26 13:09:28 on dist-test-slave-2dw5"
server_key: "e3939d3baf057195bbf5f4d507036dc7"
server_key_iv: "6f222b5c34b81e78f2c6e7a5ab4cca21"
server_key_version: "encryptionkey@0"
I20260926 13:09:28.170643 8116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:09:28.171530 8131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.171783 8116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:28.171869 8116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "ce3025b2247d423197b85ea0d89d5447"
format_stamp: "Formatted at 2026-09-26 13:09:28 on dist-test-slave-2dw5"
server_key: "e3939d3baf057195bbf5f4d507036dc7"
server_key_iv: "6f222b5c34b81e78f2c6e7a5ab4cca21"
server_key_version: "encryptionkey@0"
I20260926 13:09:28.171942 8116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:28.189373 8116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:28.191668 8116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:28.191823 8116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:28.192348 8116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:28.192392 8116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.192421 8116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:28.192438 8116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.198719 8116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37591
I20260926 13:09:28.198802 8246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37591 every 8 connection(s)
I20260926 13:09:28.199112 8116 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:28.202781 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.192842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48721 (local address 127.0.98.62:38579)
0926 13:09:28.192888 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.192890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.193101 (+ 211us) server_negotiation.cc:407] Connection header received
0926 13:09:28.193696 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.193698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.193712 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.193783 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:28.194593 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.195394 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.197113 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.197433 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.197513 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.199972 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:28.199973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:28.199973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:28.199980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:28.201263 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.201698 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.201698 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.201699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.201746 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.202030 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.202031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.202032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.202177 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:28.202339 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.202638 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.202712 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:09:28.202961 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.192832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38579 (local address 127.0.98.2:48721)
0926 13:09:28.192985 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.193022 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.193569 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.193859 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.193863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.194201 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:28.194493 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.194499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.195485 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.195486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.197021 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.197024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.197520 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.197521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.197674 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.198196 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:28.198206 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:28.199872 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:28.201344 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.201351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.201353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.201640 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.201823 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.201824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.201825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.201939 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.202359 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:28.202364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:28.202488 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.202580 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.202713 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260926 13:09:28.203485 8247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38579
I20260926 13:09:28.203558 8247 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:28.203722 8247 heartbeater.cc:511] Master 127.0.98.62:38579 requested a full tablet report, sending...
I20260926 13:09:28.204130 7904 ts_manager.cc:194] Registered new tserver with Master: ce3025b2247d423197b85ea0d89d5447 (127.0.98.2:37591)
I20260926 13:09:28.204597 7904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48721
I20260926 13:09:28.209192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 8116
I20260926 13:09:28.209301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:28.209522 392 external_mini_cluster.cc:1428] Setting key c9b9b711852f5bbf91dfdeff2d2947ed
I20260926 13:09:28.210003 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.204420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38579 (local address 127.0.98.2:32989)
0926 13:09:28.204536 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.204547 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.204603 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.204740 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.204741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.204916 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:28.204975 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.204977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.205762 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.205763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.206314 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.206316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.206677 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.206678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.206746 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.207169 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:28.207174 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:28.207586 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:28.208869 (+ 1283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.208870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.208871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.209078 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.209317 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.209318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.209318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.209392 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.209704 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:28.209705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:28.209752 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.209820 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.209926 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260926 13:09:28.210018 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.204482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32989 (local address 127.0.98.62:38579)
0926 13:09:28.204523 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.204525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.204588 (+ 63us) server_negotiation.cc:407] Connection header received
0926 13:09:28.204646 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.204647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.204656 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.204700 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:28.205042 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.205608 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.206387 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.206565 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.206614 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.207667 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:28.207669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:28.207669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:28.207675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:28.208794 (+ 1119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.209143 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.209144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.209145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.209192 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.209463 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.209463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.209464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.209640 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:28.209704 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.209854 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.209935 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 13:09:28.234380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38579
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:28.314936 8256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:28.315157 8256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:28.315183 8256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:28.316560 8256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:28.316618 8256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:28.316824 8256 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:09:28.316864 8256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:28.318993 8256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:28.319258 8256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:28.319481 8256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:28.325314 8262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:28.325348 8264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:28.325354 8261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:28.325596 8256 server_base.cc:1034] running on GCE node
I20260926 13:09:28.325831 8256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:28.326094 8256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:28.327247 8256 hybrid_clock.cc:648] HybridClock initialized: now 1790428168327224 us; error 38 us; skew 500 ppm
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:28.330523 8256 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 13:09:28.331086 8256 webserver.cc:460] Webserver started at http://127.0.98.3:36081/ using document root <none> and password file <none>
I20260926 13:09:28.331342 8256 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:28.331403 8256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:28.331534 8256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:28.332901 8256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "cc67be09a3984e979ba451801078ef63"
format_stamp: "Formatted at 2026-09-26 13:09:28 on dist-test-slave-2dw5"
server_key: "da253adeccd8df5f3c101ce026b02d5e"
server_key_iv: "c8441f1cbe9c486f84b76be31df56335"
server_key_version: "encryptionkey@0"
I20260926 13:09:28.333334 8256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cc67be09a3984e979ba451801078ef63"
format_stamp: "Formatted at 2026-09-26 13:09:28 on dist-test-slave-2dw5"
server_key: "da253adeccd8df5f3c101ce026b02d5e"
server_key_iv: "c8441f1cbe9c486f84b76be31df56335"
server_key_version: "encryptionkey@0"
I20260926 13:09:28.334941 8256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260926 13:09:28.335878 8271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.336062 8256 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:28.336169 8256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "cc67be09a3984e979ba451801078ef63"
format_stamp: "Formatted at 2026-09-26 13:09:28 on dist-test-slave-2dw5"
server_key: "da253adeccd8df5f3c101ce026b02d5e"
server_key_iv: "c8441f1cbe9c486f84b76be31df56335"
server_key_version: "encryptionkey@0"
I20260926 13:09:28.336256 8256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:28.350221 8256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:28.352875 8256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:28.353034 8256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:28.353668 8256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:28.353721 8256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.353763 8256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:28.353792 8256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428168, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:28.361384 8256 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42109
I20260926 13:09:28.361454 8386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42109 every 8 connection(s)
I20260926 13:09:28.361799 8256 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:28.364249 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.354029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44337 (local address 127.0.98.62:38579)
0926 13:09:28.354078 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.354080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.354312 (+ 232us) server_negotiation.cc:407] Connection header received
0926 13:09:28.354885 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.354887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.354900 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.354949 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:28.355713 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.356480 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.357999 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.358296 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.358337 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.361063 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:28.361065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:28.361065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:28.361073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:28.362928 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.363331 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.363332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.363333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.363391 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.363601 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.363602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.363602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.363721 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:28.363777 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.364063 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.364154 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:09:28.364315 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.354033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38579 (local address 127.0.98.3:44337)
0926 13:09:28.354184 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.354233 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.354761 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.355018 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.355023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.355314 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:28.355590 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.355596 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.356615 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.356615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.357899 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.357902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.358443 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.358443 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.358585 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.359324 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:28.359335 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:28.360973 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:28.363008 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.363015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.363017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.363265 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.363445 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.363445 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.363446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.363548 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.363777 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:28.363783 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:28.363919 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.364015 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.364149 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260926 13:09:28.365114 8387 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38579
I20260926 13:09:28.365185 8387 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:28.365414 8387 heartbeater.cc:511] Master 127.0.98.62:38579 requested a full tablet report, sending...
I20260926 13:09:28.365762 7903 ts_manager.cc:194] Registered new tserver with Master: cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109)
I20260926 13:09:28.366235 7903 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44337
I20260926 13:09:28.369163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 8256
I20260926 13:09:28.369268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:28.369467 392 external_mini_cluster.cc:1428] Setting key f00f10f4e6f2f575163a36ca0c9a0774
I20260926 13:09:28.370312 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:09:28.371701 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.366124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39119 (local address 127.0.98.62:38579)
0926 13:09:28.366168 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.366170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.366204 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:09:28.366272 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.366273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.366280 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.366331 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:28.366710 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.367335 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.368017 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.368207 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.368274 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.369147 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:28.369148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:28.369149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:28.369155 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:28.370582 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.371020 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.371021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.371021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.371074 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.371244 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.371245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.371245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.371344 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:28.371402 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.371542 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.371606 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 13:09:28.371713 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.366047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38579 (local address 127.0.98.3:39119)
0926 13:09:28.366151 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.366162 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.366219 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.366364 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.366366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.366534 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:28.366609 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.366611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.367441 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.367442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.367957 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.367958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.368295 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.368296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.368368 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.368758 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:28.368762 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:28.369053 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:28.370646 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.370647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.370648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.370799 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.371134 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.371135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.371136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.371192 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.371402 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:28.371404 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:28.371446 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.371505 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.371609 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":54,"threads_started":1}
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428168, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:09:28.392210 7972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:28.386631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.0.98.62:38579)
0926 13:09:28.386693 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.386695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.386703 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:28.386729 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.386730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.386740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.386790 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:28.387342 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.387837 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.388563 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.388743 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.388795 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.392126 (+ 3331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51958: BlockingRecv error: recv got EOF from 127.0.0.1:51958 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428168, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:28.399833 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.393009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51974 (local address 127.0.98.62:38579)
0926 13:09:28.393057 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.393059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.393070 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:28.393133 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.393133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.393142 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.393197 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:28.393667 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.394133 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.394964 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.395140 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.395191 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.397370 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:28.397372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:28.397372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:28.397412 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:28.398722 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.399074 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.399075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.399075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.399134 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.399383 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.399384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.399384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.399492 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:28.399566 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.399702 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.399761 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:09:28.401602 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:28.403432 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:28.409842 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.406178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.0.98.3:42109)
0926 13:09:28.406359 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.406364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.406375 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:28.406413 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.406415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.406430 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.406504 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:28.406671 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.407533 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.408840 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.409452 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.409511 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.409591 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.409701 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.409739 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
I20260926 13:09:28.409821 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.405867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42109 (local address 127.0.0.1:45462)
0926 13:09:28.406021 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.406064 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.406128 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.406510 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.406513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.406515 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:28.406595 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.406597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.407606 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.407607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.408739 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.408742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.409522 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.409522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.409595 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.409616 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.409678 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.409731 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":5,"thread_start_us":83,"threads_started":1}
I20260926 13:09:28.410189 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.406041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37591 (local address 127.0.0.1:45394)
0926 13:09:28.406580 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.406590 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.406637 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.406899 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.406900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.406902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:28.406934 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.406935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.408096 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.408097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.408892 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.408894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.409802 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.409802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.409859 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.409863 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.410059 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.410100 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":69,"threads_started":1}
I20260926 13:09:28.410358 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.405980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.0.98.1:32795)
0926 13:09:28.406360 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.406364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.406372 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:28.406407 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.406409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.406424 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.406493 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:28.406649 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.407970 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.409185 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.409844 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.409902 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.409973 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.410225 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.410269 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":68,"threads_started":1}
I20260926 13:09:28.410436 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.406157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.0.98.2:37591)
0926 13:09:28.406423 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.406426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.406687 (+ 261us) server_negotiation.cc:407] Connection header received
0926 13:09:28.406727 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.406729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.406743 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.406816 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:28.406992 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.408041 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.408972 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.409633 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.409697 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.409767 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.410304 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.410343 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":94,"threads_started":1}
I20260926 13:09:28.410703 8306 tablet_service.cc:1467] Processing CreateTablet for tablet 5c58d8b5519e4d63ab21c1a1dcc3f561 (DEFAULT_TABLE table=test-table [id=3cef9f0e1536417399ef38582025c870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:28.410924 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.405869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32795 (local address 127.0.0.1:36106)
0926 13:09:28.406152 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.406163 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.406222 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.406513 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.406514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.406516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:28.406586 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.406588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.408055 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.408056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.409109 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.409111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.410006 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.410007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.410078 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.410081 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.410119 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.410151 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":44,"threads_started":1}
I20260926 13:09:28.411113 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c58d8b5519e4d63ab21c1a1dcc3f561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.411058 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 5c58d8b5519e4d63ab21c1a1dcc3f561 (DEFAULT_TABLE table=test-table [id=3cef9f0e1536417399ef38582025c870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:28.411369 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c58d8b5519e4d63ab21c1a1dcc3f561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.414928 8415 tablet_bootstrap.cc:492] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63: Bootstrap starting.
I20260926 13:09:28.415078 8416 tablet_bootstrap.cc:492] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447: Bootstrap starting.
I20260926 13:09:28.415565 8415 tablet_bootstrap.cc:654] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.415571 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 5c58d8b5519e4d63ab21c1a1dcc3f561 (DEFAULT_TABLE table=test-table [id=3cef9f0e1536417399ef38582025c870]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:28.415843 8415 log.cc:824] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:28.415898 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c58d8b5519e4d63ab21c1a1dcc3f561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.416644 8415 tablet_bootstrap.cc:492] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63: No bootstrap required, opened a new log
I20260926 13:09:28.416711 8415 ts_tablet_manager.cc:1397] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:28.418339 8415 raft_consensus.cc:348] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } }
I20260926 13:09:28.418489 8415 raft_consensus.cc:374] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.418520 8415 raft_consensus.cc:729] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc67be09a3984e979ba451801078ef63, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.418641 8415 consensus_queue.cc:260] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } }
I20260926 13:09:28.418936 8415 ts_tablet_manager.cc:1428] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 13:09:28.418964 8387 heartbeater.cc:503] Master 127.0.98.62:38579 was elected leader, sending a full tablet report...
I20260926 13:09:28.419421 8419 tablet_bootstrap.cc:492] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3: Bootstrap starting.
I20260926 13:09:28.420092 8419 tablet_bootstrap.cc:654] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.420105 8416 tablet_bootstrap.cc:654] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.420439 8419 log.cc:824] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:28.420439 8416 log.cc:824] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:28.421205 8416 tablet_bootstrap.cc:492] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447: No bootstrap required, opened a new log
I20260926 13:09:28.421212 8419 tablet_bootstrap.cc:492] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3: No bootstrap required, opened a new log
I20260926 13:09:28.421284 8416 ts_tablet_manager.cc:1397] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 13:09:28.421288 8419 ts_tablet_manager.cc:1397] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:28.423342 8416 raft_consensus.cc:348] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } }
I20260926 13:09:28.423357 8419 raft_consensus.cc:348] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } }
I20260926 13:09:28.423532 8416 raft_consensus.cc:374] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.423532 8419 raft_consensus.cc:374] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.423565 8416 raft_consensus.cc:729] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce3025b2247d423197b85ea0d89d5447, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.423565 8419 raft_consensus.cc:729] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427df8ff38bf403ca17e899807721af3, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.423686 8419 consensus_queue.cc:260] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } }
I20260926 13:09:28.423686 8416 consensus_queue.cc:260] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } }
I20260926 13:09:28.423965 8416 ts_tablet_manager.cc:1428] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:09:28.423965 8419 ts_tablet_manager.cc:1428] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:09:28.424024 8107 heartbeater.cc:503] Master 127.0.98.62:38579 was elected leader, sending a full tablet report...
I20260926 13:09:28.424098 8247 heartbeater.cc:503] Master 127.0.98.62:38579 was elected leader, sending a full tablet report...
I20260926 13:09:28.434329 8423 raft_consensus.cc:482] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:28.434465 8423 raft_consensus.cc:504] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } }
I20260926 13:09:28.434795 8423 leader_election.cc:288] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109), 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795)
I20260926 13:09:28.437801 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.435022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32795 (local address 127.0.98.2:34237)
0926 13:09:28.435097 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.435108 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.435152 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.435278 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.435279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.435281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:28.435342 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.435343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.436066 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.436067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.436882 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.436885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.437509 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.437510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.437612 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.437617 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.437680 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.437727 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 13:09:28.437883 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.435093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34237 (local address 127.0.98.1:32795)
0926 13:09:28.435140 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.435142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.435152 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:28.435206 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.435207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.435215 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.435267 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:28.435422 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.435991 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.436972 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.437432 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.437485 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.437495 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.437716 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.437774 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:09:28.438006 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.435030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42109 (local address 127.0.98.2:50999)
0926 13:09:28.435330 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.435338 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.435390 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.435531 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.435532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.435533 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:28.435600 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.435602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.436254 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.436254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.437128 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.437130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.437723 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.437724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.437798 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.437802 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.437868 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.437907 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":113,"thread_start_us":115,"threads_started":1}
I20260926 13:09:28.438191 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c58d8b5519e4d63ab21c1a1dcc3f561" candidate_uuid: "ce3025b2247d423197b85ea0d89d5447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427df8ff38bf403ca17e899807721af3" is_pre_election: true
I20260926 13:09:28.438306 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.435113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50999 (local address 127.0.98.3:42109)
0926 13:09:28.435154 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.435156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.435389 (+ 233us) server_negotiation.cc:407] Connection header received
0926 13:09:28.435462 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.435464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.435477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.435518 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:28.435660 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.436173 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.437213 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.437625 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.437687 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.437696 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.438201 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.438247 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:09:28.438354 8061 raft_consensus.cc:2454] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce3025b2247d423197b85ea0d89d5447 in term 0.
I20260926 13:09:28.438611 8134 leader_election.cc:302] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427df8ff38bf403ca17e899807721af3, ce3025b2247d423197b85ea0d89d5447; no voters:
I20260926 13:09:28.438612 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c58d8b5519e4d63ab21c1a1dcc3f561" candidate_uuid: "ce3025b2247d423197b85ea0d89d5447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc67be09a3984e979ba451801078ef63" is_pre_election: true
I20260926 13:09:28.438745 8341 raft_consensus.cc:2454] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce3025b2247d423197b85ea0d89d5447 in term 0.
I20260926 13:09:28.438768 8423 raft_consensus.cc:2781] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:28.438889 8423 raft_consensus.cc:482] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:28.438936 8423 raft_consensus.cc:3037] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.439895 8423 raft_consensus.cc:504] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } }
I20260926 13:09:28.440047 8423 leader_election.cc:288] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [CANDIDATE]: Term 1 election: Requested vote from peers cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109), 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795)
I20260926 13:09:28.440223 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c58d8b5519e4d63ab21c1a1dcc3f561" candidate_uuid: "ce3025b2247d423197b85ea0d89d5447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427df8ff38bf403ca17e899807721af3"
I20260926 13:09:28.440258 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c58d8b5519e4d63ab21c1a1dcc3f561" candidate_uuid: "ce3025b2247d423197b85ea0d89d5447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc67be09a3984e979ba451801078ef63"
I20260926 13:09:28.440326 8341 raft_consensus.cc:3037] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.440340 8061 raft_consensus.cc:3037] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.441237 8341 raft_consensus.cc:2454] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce3025b2247d423197b85ea0d89d5447 in term 1.
I20260926 13:09:28.441237 8061 raft_consensus.cc:2454] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce3025b2247d423197b85ea0d89d5447 in term 1.
I20260926 13:09:28.441393 8134 leader_election.cc:302] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427df8ff38bf403ca17e899807721af3, ce3025b2247d423197b85ea0d89d5447; no voters:
I20260926 13:09:28.441473 8423 raft_consensus.cc:2781] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:28.441632 8423 raft_consensus.cc:686] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 1 LEADER]: Becoming Leader. State: Replica: ce3025b2247d423197b85ea0d89d5447, State: Running, Role: LEADER
I20260926 13:09:28.441730 8423 consensus_queue.cc:237] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } }
I20260926 13:09:28.442550 7904 catalog_manager.cc:5495] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce3025b2247d423197b85ea0d89d5447 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ce3025b2247d423197b85ea0d89d5447" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } health_report { overall_health: HEALTHY } } }
W20260926 13:09:28.449697 8248 tablet.cc:2369] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:09:28.456115 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.453747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.0.98.2:37591)
0926 13:09:28.453806 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.453808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.453816 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:28.453871 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.453873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.453925 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.453985 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:28.454118 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.454613 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.455409 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.455596 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.455649 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.455802 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.455907 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.455976 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.456013 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.456034 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:09:28.457851 8061 raft_consensus.cc:1261] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Refusing update from remote peer ce3025b2247d423197b85ea0d89d5447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:28.457860 8341 raft_consensus.cc:1261] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Refusing update from remote peer ce3025b2247d423197b85ea0d89d5447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:28.458185 8423 consensus_queue.cc:1035] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:28.458323 8425 consensus_queue.cc:1035] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:28.460546 8428 mvcc.cc:205] Tried to move back new op lower bound from 7333593778000420864 to 7333593777937874944. Current Snapshot: MvccSnapshot[applied={T|T < 7333593778000420864}]
I20260926 13:09:28.461042 8430 mvcc.cc:205] Tried to move back new op lower bound from 7333593778000420864 to 7333593777937874944. Current Snapshot: MvccSnapshot[applied={T|T < 7333593778000420864 or (T in {7333593778000420864})}]
I20260926 13:09:28.461701 8429 mvcc.cc:205] Tried to move back new op lower bound from 7333593778000420864 to 7333593777937874944. Current Snapshot: MvccSnapshot[applied={T|T < 7333593778000420864 or (T in {7333593778000420864})}]
I20260926 13:09:28.462692 7903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:28.462793 7903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:28.463541 7903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc: Sending DeleteTablet for 3 replicas of tablet 5c58d8b5519e4d63ab21c1a1dcc3f561
I20260926 13:09:28.463840 8306 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c58d8b5519e4d63ab21c1a1dcc3f561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45462
I20260926 13:09:28.463900 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c58d8b5519e4d63ab21c1a1dcc3f561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45394
I20260926 13:09:28.464066 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c58d8b5519e4d63ab21c1a1dcc3f561 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36106
I20260926 13:09:28.464020 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:28.464182 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:28.464285 8438 tablet_replica.cc:330] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63: stopping tablet replica
I20260926 13:09:28.464378 8438 raft_consensus.cc:2229] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:28.464511 8438 raft_consensus.cc:2258] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:28.465157 8438 ts_tablet_manager.cc:1905] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:28.465394 8306 tablet_service.cc:1467] Processing CreateTablet for tablet a7be4aac94de414b8404130c7e2ae376 (DEFAULT_TABLE table=test-table [id=4f322db9bd0f40c787c8a8b8ac70e6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:28.465431 8041 tablet_service.cc:1467] Processing CreateTablet for tablet a7be4aac94de414b8404130c7e2ae376 (DEFAULT_TABLE table=test-table [id=4f322db9bd0f40c787c8a8b8ac70e6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:28.465502 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7be4aac94de414b8404130c7e2ae376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.465531 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7be4aac94de414b8404130c7e2ae376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.466835 8415 tablet_bootstrap.cc:492] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63: Bootstrap starting.
I20260926 13:09:28.464378 8439 tablet_replica.cc:330] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447: stopping tablet replica
I20260926 13:09:28.467087 8439 raft_consensus.cc:2229] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:09:28.467187 8439 raft_consensus.cc:2258] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:28.467245 8415 tablet_bootstrap.cc:654] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.467305 8181 tablet_service.cc:1467] Processing CreateTablet for tablet a7be4aac94de414b8404130c7e2ae376 (DEFAULT_TABLE table=test-table [id=4f322db9bd0f40c787c8a8b8ac70e6cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:28.467407 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7be4aac94de414b8404130c7e2ae376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.467643 8439 ts_tablet_manager.cc:1905] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:28.467773 8415 tablet_bootstrap.cc:492] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63: No bootstrap required, opened a new log
I20260926 13:09:28.467808 8415 ts_tablet_manager.cc:1397] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:28.467929 8415 raft_consensus.cc:348] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.467990 8415 raft_consensus.cc:374] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.468008 8415 raft_consensus.cc:729] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc67be09a3984e979ba451801078ef63, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.468048 8415 consensus_queue.cc:260] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.468124 8415 ts_tablet_manager.cc:1428] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.468658 8416 tablet_bootstrap.cc:492] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447: Bootstrap starting.
I20260926 13:09:28.468734 8419 tablet_bootstrap.cc:492] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3: Bootstrap starting.
I20260926 13:09:28.469044 8416 tablet_bootstrap.cc:654] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.469081 8439 ts_tablet_manager.cc:1918] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:28.469128 8419 tablet_bootstrap.cc:654] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.469144 8439 log.cc:1196] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/5c58d8b5519e4d63ab21c1a1dcc3f561
I20260926 13:09:28.469393 8439 ts_tablet_manager.cc:1939] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P ce3025b2247d423197b85ea0d89d5447: Deleting consensus metadata
I20260926 13:09:28.469753 8419 tablet_bootstrap.cc:492] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3: No bootstrap required, opened a new log
I20260926 13:09:28.469803 8419 ts_tablet_manager.cc:1397] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:28.469950 8419 raft_consensus.cc:348] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.470031 8419 raft_consensus.cc:374] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.470052 8419 raft_consensus.cc:729] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427df8ff38bf403ca17e899807721af3, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.470093 8419 consensus_queue.cc:260] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.470180 8419 ts_tablet_manager.cc:1428] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:28.470196 8416 tablet_bootstrap.cc:492] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447: No bootstrap required, opened a new log
I20260926 13:09:28.470234 8416 ts_tablet_manager.cc:1397] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:28.470377 8416 raft_consensus.cc:348] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.470432 8416 raft_consensus.cc:374] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.470450 8416 raft_consensus.cc:729] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce3025b2247d423197b85ea0d89d5447, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.470489 8416 consensus_queue.cc:260] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.470567 8416 ts_tablet_manager.cc:1428] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.470942 8438 ts_tablet_manager.cc:1918] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:28.471024 8438 log.cc:1196] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/5c58d8b5519e4d63ab21c1a1dcc3f561
I20260926 13:09:28.471251 8440 tablet_replica.cc:330] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3: stopping tablet replica
I20260926 13:09:28.471275 8438 ts_tablet_manager.cc:1939] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P cc67be09a3984e979ba451801078ef63: Deleting consensus metadata
I20260926 13:09:28.471242 7890 catalog_manager.cc:4841] TS ce3025b2247d423197b85ea0d89d5447 (127.0.98.2:37591): tablet 5c58d8b5519e4d63ab21c1a1dcc3f561 (table test-table [id=3cef9f0e1536417399ef38582025c870]) successfully deleted
I20260926 13:09:28.471385 8440 raft_consensus.cc:2229] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:28.471499 8440 raft_consensus.cc:2258] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:28.471699 7891 catalog_manager.cc:4841] TS cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109): tablet 5c58d8b5519e4d63ab21c1a1dcc3f561 (table test-table [id=3cef9f0e1536417399ef38582025c870]) successfully deleted
I20260926 13:09:28.471938 8440 ts_tablet_manager.cc:1905] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:28.473670 8440 ts_tablet_manager.cc:1918] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:28.473744 8440 log.cc:1196] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/wal/wals/5c58d8b5519e4d63ab21c1a1dcc3f561
I20260926 13:09:28.473964 8440 ts_tablet_manager.cc:1939] T 5c58d8b5519e4d63ab21c1a1dcc3f561 P 427df8ff38bf403ca17e899807721af3: Deleting consensus metadata
I20260926 13:09:28.474313 7890 catalog_manager.cc:4841] TS 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795): tablet 5c58d8b5519e4d63ab21c1a1dcc3f561 (table test-table [id=3cef9f0e1536417399ef38582025c870]) successfully deleted
I20260926 13:09:28.482545 8422 raft_consensus.cc:482] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:28.482631 8422 raft_consensus.cc:504] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.482983 8422 leader_election.cc:288] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce3025b2247d423197b85ea0d89d5447 (127.0.98.2:37591), cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109)
I20260926 13:09:28.486232 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.483125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42109 (local address 127.0.98.1:57573)
0926 13:09:28.483184 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.483195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.483242 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.483422 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.483423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.483424 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:28.483506 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.483508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.484417 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.484418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.485475 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.485477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.485984 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.485985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.486055 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.486059 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.486108 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.486150 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 13:09:28.486222 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.483184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57573 (local address 127.0.98.3:42109)
0926 13:09:28.483224 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.483226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.483235 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:28.483328 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.483329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.483338 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.483380 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:28.483579 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.484333 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.485543 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.485918 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.485976 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.485986 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.486120 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.486158 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:09:28.486487 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.483296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60285 (local address 127.0.98.2:37591)
0926 13:09:28.483348 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.483350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.483358 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:28.483383 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.483384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.483392 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.483443 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:28.483600 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.484067 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.485512 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.486092 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.486161 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.486171 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.486379 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.486417 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:09:28.486531 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7be4aac94de414b8404130c7e2ae376" candidate_uuid: "427df8ff38bf403ca17e899807721af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc67be09a3984e979ba451801078ef63" is_pre_election: true
I20260926 13:09:28.486611 8341 raft_consensus.cc:2454] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427df8ff38bf403ca17e899807721af3 in term 0.
I20260926 13:09:28.486709 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.483137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37591 (local address 127.0.98.1:60285)
0926 13:09:28.483274 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.483282 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.483323 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.483451 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.483451 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.483452 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:28.483514 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.483515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.484146 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.484147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.485429 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.485432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.486205 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.486206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.486286 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.486290 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.486591 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.486635 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260926 13:09:28.486776 7995 leader_election.cc:302] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427df8ff38bf403ca17e899807721af3, cc67be09a3984e979ba451801078ef63; no voters:
I20260926 13:09:28.486945 8422 raft_consensus.cc:2781] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:28.486979 8422 raft_consensus.cc:482] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:28.486995 8422 raft_consensus.cc:3037] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.487000 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7be4aac94de414b8404130c7e2ae376" candidate_uuid: "427df8ff38bf403ca17e899807721af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce3025b2247d423197b85ea0d89d5447" is_pre_election: true
I20260926 13:09:28.487106 8201 raft_consensus.cc:2454] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 427df8ff38bf403ca17e899807721af3 in term 0.
I20260926 13:09:28.487730 8422 raft_consensus.cc:504] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.487841 8422 leader_election.cc:288] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [CANDIDATE]: Term 1 election: Requested vote from peers ce3025b2247d423197b85ea0d89d5447 (127.0.98.2:37591), cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109)
I20260926 13:09:28.487975 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7be4aac94de414b8404130c7e2ae376" candidate_uuid: "427df8ff38bf403ca17e899807721af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce3025b2247d423197b85ea0d89d5447"
I20260926 13:09:28.488044 8201 raft_consensus.cc:3037] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.488080 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7be4aac94de414b8404130c7e2ae376" candidate_uuid: "427df8ff38bf403ca17e899807721af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc67be09a3984e979ba451801078ef63"
I20260926 13:09:28.488143 8341 raft_consensus.cc:3037] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.488822 8201 raft_consensus.cc:2454] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427df8ff38bf403ca17e899807721af3 in term 1.
I20260926 13:09:28.488822 8341 raft_consensus.cc:2454] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 427df8ff38bf403ca17e899807721af3 in term 1.
I20260926 13:09:28.488976 7995 leader_election.cc:302] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427df8ff38bf403ca17e899807721af3, cc67be09a3984e979ba451801078ef63; no voters:
I20260926 13:09:28.489064 8422 raft_consensus.cc:2781] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:28.489171 8422 raft_consensus.cc:686] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 1 LEADER]: Becoming Leader. State: Replica: 427df8ff38bf403ca17e899807721af3, State: Running, Role: LEADER
I20260926 13:09:28.489233 8422 consensus_queue.cc:237] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.489876 7903 catalog_manager.cc:5495] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 427df8ff38bf403ca17e899807721af3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "427df8ff38bf403ca17e899807721af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } health_report { overall_health: UNKNOWN } } }
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428167, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:28.527212 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.519590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.0.98.62:38579)
0926 13:09:28.519646 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.519648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.519746 (+ 98us) server_negotiation.cc:407] Connection header received
0926 13:09:28.519792 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.519793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.519804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.519878 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:28.520346 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.521107 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.521950 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.522243 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.522324 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.524790 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:28.524791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:28.524792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:28.524799 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:28.526046 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.526482 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.526483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.526483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.526535 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.526732 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.526733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.526733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.526834 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:28.526920 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.527073 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.527142 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 13:09:28.527329 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.519526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38579 (local address 127.0.0.1:51990)
0926 13:09:28.519660 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.519684 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.519745 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.519883 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.519883 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.520206 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:28.520259 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.520262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.521223 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.521224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.521867 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.521870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.522387 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.522388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.522433 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.523105 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:28.523110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:28.524686 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:28.526132 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.526140 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.526142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.526412 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.526597 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:28.526598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:28.526599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:28.526670 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:28.526913 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:28.526921 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:28.526977 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.527055 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.527216 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":38,"thread_start_us":71,"threads_started":1}
I20260926 13:09:28.529606 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51990:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 13:09:28.529829 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:28.531157 8306 tablet_service.cc:1467] Processing CreateTablet for tablet 778c810da8af4a5eaa3f0bc819870993 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72cbb78785f6443fba3bd7525c0ad580]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:09:28.531169 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 778c810da8af4a5eaa3f0bc819870993 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72cbb78785f6443fba3bd7525c0ad580]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:09:28.531207 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 778c810da8af4a5eaa3f0bc819870993 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72cbb78785f6443fba3bd7525c0ad580]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:09:28.531319 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778c810da8af4a5eaa3f0bc819870993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.531317 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778c810da8af4a5eaa3f0bc819870993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.531325 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 778c810da8af4a5eaa3f0bc819870993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.532778 8416 tablet_bootstrap.cc:492] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447: Bootstrap starting.
I20260926 13:09:28.532784 8419 tablet_bootstrap.cc:492] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3: Bootstrap starting.
I20260926 13:09:28.533175 8416 tablet_bootstrap.cc:654] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.533175 8419 tablet_bootstrap.cc:654] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.533519 8415 tablet_bootstrap.cc:492] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: Bootstrap starting.
I20260926 13:09:28.533923 8415 tablet_bootstrap.cc:654] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.534336 8419 tablet_bootstrap.cc:492] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3: No bootstrap required, opened a new log
I20260926 13:09:28.534386 8419 ts_tablet_manager.cc:1397] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:28.534435 8416 tablet_bootstrap.cc:492] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447: No bootstrap required, opened a new log
I20260926 13:09:28.534472 8416 ts_tablet_manager.cc:1397] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:28.534525 8419 raft_consensus.cc:348] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.534595 8419 raft_consensus.cc:374] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.534612 8419 raft_consensus.cc:729] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427df8ff38bf403ca17e899807721af3, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.534602 8416 raft_consensus.cc:348] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.534665 8416 raft_consensus.cc:374] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.534660 8419 consensus_queue.cc:260] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.534685 8416 raft_consensus.cc:729] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce3025b2247d423197b85ea0d89d5447, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.534720 8416 consensus_queue.cc:260] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.534760 8419 ts_tablet_manager.cc:1428] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.534799 8416 ts_tablet_manager.cc:1428] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.534803 8422 tablet_replica.cc:439] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260926 13:09:28.534920 8422 tablet_replica.cc:442] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:28.535004 8415 tablet_bootstrap.cc:492] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: No bootstrap required, opened a new log
I20260926 13:09:28.535043 8415 ts_tablet_manager.cc:1397] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:28.534997 8433 tablet_replica.cc:439] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260926 13:09:28.535072 8433 tablet_replica.cc:442] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:28.535175 8415 raft_consensus.cc:348] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.535231 8415 raft_consensus.cc:374] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.535248 8415 raft_consensus.cc:729] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc67be09a3984e979ba451801078ef63, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.535286 8415 consensus_queue.cc:260] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.535360 8415 ts_tablet_manager.cc:1428] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.535444 8418 tablet_replica.cc:439] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260926 13:09:28.535550 8418 tablet_replica.cc:442] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 13:09:28.543146 8108 tablet.cc:2369] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:09:28.559902 8418 raft_consensus.cc:482] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:28.560009 8418 raft_consensus.cc:504] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.560340 8418 leader_election.cc:288] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795), ce3025b2247d423197b85ea0d89d5447 (127.0.98.2:37591)
I20260926 13:09:28.564106 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.560493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32795 (local address 127.0.98.3:45535)
0926 13:09:28.560534 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.560545 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.560587 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.560733 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.560734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.560735 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:28.560808 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.560810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.561685 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.561686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.562933 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.562937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.563838 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.563839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.563917 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.563922 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.563978 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.564023 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 13:09:28.564111 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.560560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45535 (local address 127.0.98.1:32795)
0926 13:09:28.560605 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.560607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.560619 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:09:28.560646 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.560647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.560657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.560722 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:28.560892 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.561598 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.563029 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.563747 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.563817 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.563834 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.563992 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.564041 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:09:28.564131 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.560832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32941 (local address 127.0.98.2:37591)
0926 13:09:28.560874 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.560876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.560939 (+ 63us) server_negotiation.cc:407] Connection header received
0926 13:09:28.560963 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.560964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.560972 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.561017 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:28.561226 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.561828 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.563296 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.563798 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.563834 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.563842 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.564041 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.564078 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:09:28.564358 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.560587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37591 (local address 127.0.98.3:32941)
0926 13:09:28.560847 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.560857 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.560915 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.561033 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.561034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.561035 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:28.561109 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.561110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.561913 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.561914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.563206 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.563208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.563895 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.563896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.563966 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.563970 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.564233 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.564275 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":138,"threads_started":1}
I20260926 13:09:28.564368 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778c810da8af4a5eaa3f0bc819870993" candidate_uuid: "cc67be09a3984e979ba451801078ef63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427df8ff38bf403ca17e899807721af3" is_pre_election: true
I20260926 13:09:28.564460 8061 raft_consensus.cc:2454] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc67be09a3984e979ba451801078ef63 in term 0.
I20260926 13:09:28.564563 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778c810da8af4a5eaa3f0bc819870993" candidate_uuid: "cc67be09a3984e979ba451801078ef63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce3025b2247d423197b85ea0d89d5447" is_pre_election: true
I20260926 13:09:28.564640 8201 raft_consensus.cc:2454] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc67be09a3984e979ba451801078ef63 in term 0.
I20260926 13:09:28.564620 8274 leader_election.cc:302] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427df8ff38bf403ca17e899807721af3, cc67be09a3984e979ba451801078ef63; no voters:
I20260926 13:09:28.564787 8418 raft_consensus.cc:2781] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:28.564849 8418 raft_consensus.cc:482] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:28.564877 8418 raft_consensus.cc:3037] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.565531 8418 raft_consensus.cc:504] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.565652 8418 leader_election.cc:288] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [CANDIDATE]: Term 1 election: Requested vote from peers 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795), ce3025b2247d423197b85ea0d89d5447 (127.0.98.2:37591)
I20260926 13:09:28.565850 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778c810da8af4a5eaa3f0bc819870993" candidate_uuid: "cc67be09a3984e979ba451801078ef63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce3025b2247d423197b85ea0d89d5447"
I20260926 13:09:28.565886 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "778c810da8af4a5eaa3f0bc819870993" candidate_uuid: "cc67be09a3984e979ba451801078ef63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427df8ff38bf403ca17e899807721af3"
I20260926 13:09:28.565932 8201 raft_consensus.cc:3037] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.565960 8061 raft_consensus.cc:3037] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.566468 8201 raft_consensus.cc:2454] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc67be09a3984e979ba451801078ef63 in term 1.
I20260926 13:09:28.566627 8274 leader_election.cc:302] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc67be09a3984e979ba451801078ef63, ce3025b2247d423197b85ea0d89d5447; no voters:
I20260926 13:09:28.566705 8418 raft_consensus.cc:2781] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:28.566701 8061 raft_consensus.cc:2454] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc67be09a3984e979ba451801078ef63 in term 1.
I20260926 13:09:28.566869 8418 raft_consensus.cc:686] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [term 1 LEADER]: Becoming Leader. State: Replica: cc67be09a3984e979ba451801078ef63, State: Running, Role: LEADER
I20260926 13:09:28.566948 8418 consensus_queue.cc:237] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.567308 8457 tablet_replica.cc:439] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: TxnStatusTablet state changed. Reason: New leader cc67be09a3984e979ba451801078ef63. Latest consensus state: current_term: 1 leader_uuid: "cc67be09a3984e979ba451801078ef63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260926 13:09:28.567380 8457 tablet_replica.cc:442] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:09:28.567528 8459 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 13:09:28.567597 8459 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 13:09:28.567607 7904 catalog_manager.cc:5495] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc67be09a3984e979ba451801078ef63 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cc67be09a3984e979ba451801078ef63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } health_report { overall_health: HEALTHY } } }
I20260926 13:09:28.576751 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.573481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.0.98.3:42109)
0926 13:09:28.573527 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.573529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.573705 (+ 176us) server_negotiation.cc:407] Connection header received
0926 13:09:28.573788 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.573789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.573829 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.573892 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:28.574003 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.574739 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.575843 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.576136 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.576194 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.576404 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.576531 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.576597 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.576647 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.576666 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:09:28.576781 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.573568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42109 (local address 127.0.0.1:45474)
0926 13:09:28.573636 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.573649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.573708 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.573900 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.573901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.573902 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:28.573954 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.573957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.574811 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.574812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.575732 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.575754 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.576210 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.576211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.576290 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.576311 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.576592 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.576598 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.576644 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.576698 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 13:09:28.578128 8061 raft_consensus.cc:1261] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Refusing update from remote peer cc67be09a3984e979ba451801078ef63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:28.578171 8201 raft_consensus.cc:1261] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Refusing update from remote peer cc67be09a3984e979ba451801078ef63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:28.578540 8418 consensus_queue.cc:1035] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:28.578692 8457 consensus_queue.cc:1035] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:28.579270 8422 tablet_replica.cc:439] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3: TxnStatusTablet state changed. Reason: New leader cc67be09a3984e979ba451801078ef63. Latest consensus state: current_term: 1 leader_uuid: "cc67be09a3984e979ba451801078ef63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260926 13:09:28.579289 8433 tablet_replica.cc:439] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447: TxnStatusTablet state changed. Reason: New leader cc67be09a3984e979ba451801078ef63. Latest consensus state: current_term: 1 leader_uuid: "cc67be09a3984e979ba451801078ef63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260926 13:09:28.579365 8422 tablet_replica.cc:442] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:28.579370 8433 tablet_replica.cc:442] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:28.580667 8418 tablet_replica.cc:439] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc67be09a3984e979ba451801078ef63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260926 13:09:28.580750 8418 tablet_replica.cc:442] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:09:28.580880 8422 tablet_replica.cc:439] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc67be09a3984e979ba451801078ef63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260926 13:09:28.580958 8422 tablet_replica.cc:442] T 778c810da8af4a5eaa3f0bc819870993 P 427df8ff38bf403ca17e899807721af3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:28.585422 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.582445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.0.98.1:32795)
0926 13:09:28.582481 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.582483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.582492 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:28.582552 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.582553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.582577 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.582652 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:28.582773 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.583522 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.584558 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.584836 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.584905 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.585072 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.585196 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.585269 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.585311 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.585344 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:09:28.587800 8457 tablet_replica.cc:439] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc67be09a3984e979ba451801078ef63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260926 13:09:28.588025 8457 tablet_replica.cc:442] T 778c810da8af4a5eaa3f0bc819870993 P cc67be09a3984e979ba451801078ef63: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:09:28.588443 8433 tablet_replica.cc:439] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc67be09a3984e979ba451801078ef63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260926 13:09:28.588532 8433 tablet_replica.cc:442] T 778c810da8af4a5eaa3f0bc819870993 P ce3025b2247d423197b85ea0d89d5447: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:09:28.591905 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.588098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35089 (local address 127.0.98.3:42109)
0926 13:09:28.588137 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.588139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.588988 (+ 849us) server_negotiation.cc:407] Connection header received
0926 13:09:28.589045 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.589046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.589056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.589098 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:28.589225 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.589968 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.591145 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.591399 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.591460 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.591684 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.591717 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.591759 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.591813 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.591835 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:09:28.592144 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.587939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42109 (local address 127.0.98.1:35089)
0926 13:09:28.588931 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.588943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.588999 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.589108 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.589109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.589110 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:28.589172 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.589173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.590045 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.590046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.591046 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.591049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.591479 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.591480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.591591 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.591610 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.591770 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.591771 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.591811 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.591853 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"thread_start_us":74,"threads_started":1}
I20260926 13:09:28.594447 8341 raft_consensus.cc:1261] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Refusing update from remote peer 427df8ff38bf403ca17e899807721af3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:28.594461 8201 raft_consensus.cc:1261] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Refusing update from remote peer 427df8ff38bf403ca17e899807721af3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:28.594933 8445 consensus_queue.cc:1035] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:28.595052 8422 consensus_queue.cc:1035] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:28.603583 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.600542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52479 (local address 127.0.98.1:32795)
0926 13:09:28.600694 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.600695 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.600700 (+ 5us) server_negotiation.cc:407] Connection header received
0926 13:09:28.600716 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.600716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.600724 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.600759 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:28.600987 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.601552 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.602678 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.603025 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.603094 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.603264 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.603301 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.603432 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.603481 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.603502 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260926 13:09:28.603590 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.600516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32795 (local address 127.0.98.3:52479)
0926 13:09:28.600589 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.600600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.600636 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.600786 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.600787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.600789 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:28.600850 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.600852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.601621 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.601622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.602570 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.602573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.603091 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.603092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.603161 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.603192 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.603436 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.603438 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.603479 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.603523 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 13:09:28.613128 7903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:28.613237 7903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:28.613790 7903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc: Sending DeleteTablet for 3 replicas of tablet a7be4aac94de414b8404130c7e2ae376
I20260926 13:09:28.614032 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet a7be4aac94de414b8404130c7e2ae376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36106
I20260926 13:09:28.614086 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet a7be4aac94de414b8404130c7e2ae376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45394
I20260926 13:09:28.614136 8440 tablet_replica.cc:330] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3: stopping tablet replica
I20260926 13:09:28.614193 8440 raft_consensus.cc:2229] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:09:28.614276 8440 raft_consensus.cc:2258] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:28.614363 8439 tablet_replica.cc:330] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447: stopping tablet replica
I20260926 13:09:28.614418 8439 raft_consensus.cc:2229] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:28.614449 8439 raft_consensus.cc:2258] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:28.614634 8440 ts_tablet_manager.cc:1905] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:28.614635 8306 tablet_service.cc:1514] Processing DeleteTablet for tablet a7be4aac94de414b8404130c7e2ae376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45462
I20260926 13:09:28.616192 8438 tablet_replica.cc:330] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63: stopping tablet replica
I20260926 13:09:28.616255 8438 raft_consensus.cc:2229] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:28.616289 8438 raft_consensus.cc:2258] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:28.616573 8440 ts_tablet_manager.cc:1918] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:28.616624 8440 log.cc:1196] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/wal/wals/a7be4aac94de414b8404130c7e2ae376
I20260926 13:09:28.616631 8438 ts_tablet_manager.cc:1905] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:28.616868 8440 ts_tablet_manager.cc:1939] T a7be4aac94de414b8404130c7e2ae376 P 427df8ff38bf403ca17e899807721af3: Deleting consensus metadata
I20260926 13:09:28.617302 7890 catalog_manager.cc:4841] TS 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795): tablet a7be4aac94de414b8404130c7e2ae376 (table test-table [id=4f322db9bd0f40c787c8a8b8ac70e6cc]) successfully deleted
I20260926 13:09:28.617877 8439 ts_tablet_manager.cc:1905] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:28.618407 8438 ts_tablet_manager.cc:1918] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:28.618458 8438 log.cc:1196] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/a7be4aac94de414b8404130c7e2ae376
I20260926 13:09:28.618651 8438 ts_tablet_manager.cc:1939] T a7be4aac94de414b8404130c7e2ae376 P cc67be09a3984e979ba451801078ef63: Deleting consensus metadata
I20260926 13:09:28.619136 7891 catalog_manager.cc:4841] TS cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109): tablet a7be4aac94de414b8404130c7e2ae376 (table test-table [id=4f322db9bd0f40c787c8a8b8ac70e6cc]) successfully deleted
I20260926 13:09:28.619458 8439 ts_tablet_manager.cc:1918] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:28.619509 8439 log.cc:1196] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/a7be4aac94de414b8404130c7e2ae376
I20260926 13:09:28.619678 8439 ts_tablet_manager.cc:1939] T a7be4aac94de414b8404130c7e2ae376 P ce3025b2247d423197b85ea0d89d5447: Deleting consensus metadata
I20260926 13:09:28.620126 7890 catalog_manager.cc:4841] TS ce3025b2247d423197b85ea0d89d5447 (127.0.98.2:37591): tablet a7be4aac94de414b8404130c7e2ae376 (table test-table [id=4f322db9bd0f40c787c8a8b8ac70e6cc]) successfully deleted
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:09:28.647740 7972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:28.632106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.0.98.62:38579)
0926 13:09:28.632154 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.632156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.632275 (+ 119us) server_negotiation.cc:407] Connection header received
0926 13:09:28.632356 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.632357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.632368 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.632407 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:28.632952 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.633442 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.634162 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.634374 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.634430 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.647684 (+ 13254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51998: BlockingRecv error: recv got EOF from 127.0.0.1:51998 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:09:28.655616 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.648686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.0.98.62:38579)
0926 13:09:28.648730 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.648732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.648940 (+ 208us) server_negotiation.cc:407] Connection header received
0926 13:09:28.649004 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.649005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.649016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.649058 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:28.649511 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.650015 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.650691 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.650940 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.651004 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.653023 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:28.653025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:28.653025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:28.653034 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:28.654393 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.654805 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.654806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.654807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.654879 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:28.655083 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:28.655084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:28.655084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:28.655212 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:28.655279 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.655450 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.655523 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:09:28.657045 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:28.657198 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:28.658459 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 27d564aeb1a04dcaad076e96a4cd90b3 (DEFAULT_TABLE table=test-table [id=552d76d8eeb5455cb136f426a572ae8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:28.658459 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 27d564aeb1a04dcaad076e96a4cd90b3 (DEFAULT_TABLE table=test-table [id=552d76d8eeb5455cb136f426a572ae8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:28.658588 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d564aeb1a04dcaad076e96a4cd90b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.658588 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d564aeb1a04dcaad076e96a4cd90b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.659898 8416 tablet_bootstrap.cc:492] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447: Bootstrap starting.
I20260926 13:09:28.659933 8306 tablet_service.cc:1467] Processing CreateTablet for tablet 27d564aeb1a04dcaad076e96a4cd90b3 (DEFAULT_TABLE table=test-table [id=552d76d8eeb5455cb136f426a572ae8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:28.660034 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d564aeb1a04dcaad076e96a4cd90b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.660208 8416 tablet_bootstrap.cc:654] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.661427 8415 tablet_bootstrap.cc:492] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63: Bootstrap starting.
I20260926 13:09:28.661470 8419 tablet_bootstrap.cc:492] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: Bootstrap starting.
I20260926 13:09:28.661891 8415 tablet_bootstrap.cc:654] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.661890 8419 tablet_bootstrap.cc:654] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.662693 8415 tablet_bootstrap.cc:492] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63: No bootstrap required, opened a new log
I20260926 13:09:28.662755 8415 ts_tablet_manager.cc:1397] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:28.662927 8415 raft_consensus.cc:348] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.662998 8415 raft_consensus.cc:374] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.663017 8415 raft_consensus.cc:729] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc67be09a3984e979ba451801078ef63, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.663058 8415 consensus_queue.cc:260] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.663156 8415 ts_tablet_manager.cc:1428] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.663189 8419 tablet_bootstrap.cc:492] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: No bootstrap required, opened a new log
I20260926 13:09:28.663241 8419 ts_tablet_manager.cc:1397] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:28.663378 8419 raft_consensus.cc:348] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.663445 8419 raft_consensus.cc:374] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.663461 8419 raft_consensus.cc:729] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427df8ff38bf403ca17e899807721af3, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.663498 8419 consensus_queue.cc:260] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.663928 8419 ts_tablet_manager.cc:1428] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:09:28.664623 8416 tablet_bootstrap.cc:492] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447: No bootstrap required, opened a new log
I20260926 13:09:28.664681 8416 ts_tablet_manager.cc:1397] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 13:09:28.664803 8416 raft_consensus.cc:348] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.664861 8416 raft_consensus.cc:374] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.664877 8416 raft_consensus.cc:729] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce3025b2247d423197b85ea0d89d5447, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.664915 8416 consensus_queue.cc:260] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.665012 8416 ts_tablet_manager.cc:1428] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.727895 8433 raft_consensus.cc:482] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:28.728039 8433 raft_consensus.cc:504] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.728204 8433 leader_election.cc:288] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795), cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109)
I20260926 13:09:28.728474 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d564aeb1a04dcaad076e96a4cd90b3" candidate_uuid: "ce3025b2247d423197b85ea0d89d5447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc67be09a3984e979ba451801078ef63" is_pre_election: true
I20260926 13:09:28.728498 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d564aeb1a04dcaad076e96a4cd90b3" candidate_uuid: "ce3025b2247d423197b85ea0d89d5447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427df8ff38bf403ca17e899807721af3" is_pre_election: true
I20260926 13:09:28.728564 8341 raft_consensus.cc:2454] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce3025b2247d423197b85ea0d89d5447 in term 0.
I20260926 13:09:28.728574 8061 raft_consensus.cc:2454] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce3025b2247d423197b85ea0d89d5447 in term 0.
I20260926 13:09:28.728763 8134 leader_election.cc:302] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427df8ff38bf403ca17e899807721af3, ce3025b2247d423197b85ea0d89d5447; no voters:
I20260926 13:09:28.728868 8433 raft_consensus.cc:2781] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:28.728905 8433 raft_consensus.cc:482] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:28.728924 8433 raft_consensus.cc:3037] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.729712 8433 raft_consensus.cc:504] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.729838 8433 leader_election.cc:288] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [CANDIDATE]: Term 1 election: Requested vote from peers 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795), cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109)
I20260926 13:09:28.730019 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d564aeb1a04dcaad076e96a4cd90b3" candidate_uuid: "ce3025b2247d423197b85ea0d89d5447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427df8ff38bf403ca17e899807721af3"
I20260926 13:09:28.730093 8061 raft_consensus.cc:3037] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.730064 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d564aeb1a04dcaad076e96a4cd90b3" candidate_uuid: "ce3025b2247d423197b85ea0d89d5447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc67be09a3984e979ba451801078ef63"
I20260926 13:09:28.730121 8341 raft_consensus.cc:3037] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.730866 8061 raft_consensus.cc:2454] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce3025b2247d423197b85ea0d89d5447 in term 1.
I20260926 13:09:28.730866 8341 raft_consensus.cc:2454] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce3025b2247d423197b85ea0d89d5447 in term 1.
I20260926 13:09:28.731036 8134 leader_election.cc:302] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427df8ff38bf403ca17e899807721af3, ce3025b2247d423197b85ea0d89d5447; no voters:
I20260926 13:09:28.731160 8433 raft_consensus.cc:2781] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:28.731204 8433 raft_consensus.cc:686] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 1 LEADER]: Becoming Leader. State: Replica: ce3025b2247d423197b85ea0d89d5447, State: Running, Role: LEADER
I20260926 13:09:28.731246 8433 consensus_queue.cc:237] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.731832 7903 catalog_manager.cc:5495] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce3025b2247d423197b85ea0d89d5447 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ce3025b2247d423197b85ea0d89d5447" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } health_report { overall_health: UNKNOWN } } }
I20260926 13:09:28.738755 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.736535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.0.98.2:37591)
0926 13:09:28.736575 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.736577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.736585 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:28.736644 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.736645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.736654 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.736698 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:28.736832 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.737327 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.738095 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.738314 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.738368 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.738516 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.738551 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.738593 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.738644 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.738667 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:09:28.739490 8341 raft_consensus.cc:1261] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Refusing update from remote peer ce3025b2247d423197b85ea0d89d5447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:28.739645 8061 raft_consensus.cc:1261] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Refusing update from remote peer ce3025b2247d423197b85ea0d89d5447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:28.739848 8423 consensus_queue.cc:1035] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:28.739957 8433 consensus_queue.cc:1035] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:28.742751 7903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:28.742874 7903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:28.743582 7903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc: Sending DeleteTablet for 3 replicas of tablet 27d564aeb1a04dcaad076e96a4cd90b3
I20260926 13:09:28.743978 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d564aeb1a04dcaad076e96a4cd90b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36106
I20260926 13:09:28.744029 8306 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d564aeb1a04dcaad076e96a4cd90b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45462
I20260926 13:09:28.744069 8440 tablet_replica.cc:330] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: stopping tablet replica
I20260926 13:09:28.744103 8438 tablet_replica.cc:330] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63: stopping tablet replica
I20260926 13:09:28.744113 8440 raft_consensus.cc:2229] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:28.744141 8438 raft_consensus.cc:2229] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:28.744156 8440 pending_rounds.cc:62] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: Trying to abort 2 pending ops.
I20260926 13:09:28.744190 8438 raft_consensus.cc:2258] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:28.744174 8440 pending_rounds.cc:68] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333593779155304448 op_type: WRITE_OP write_request { tablet_id: "27d564aeb1a04dcaad076e96a4cd90b3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\264\212\337\325\006\"8\n\ntest-admin\022*\n 552d76d8eeb5455cb136f426a572ae8f\020\001\030\001 \001(\001" signature: "<redacted>""H\307\r>C\311\222;j\023\227\351\025\3718\223\271[\251I\300\000\273\272\300\254\301h\205\334\"%\335^\030\037\\j\330\"G{\311\200\tJn\334\250\316\361\322\347\305\\\227\321\371#\323\313\002\353h" signing_key_seq_num: 0 } } request_id { client_id: "9c2f4bd639c84097a063b9f49d704d35" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260926 13:09:28.744359 8440 pending_rounds.cc:68] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333593779123572736 op_type: NO_OP noop_request { }
I20260926 13:09:28.744401 8440 raft_consensus.cc:2866] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 13:09:28.744427 8440 raft_consensus.cc:2258] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 13:09:28.744752 8134 consensus_peers.cc:487] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 -> Peer 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795): Couldn't send request to peer 427df8ff38bf403ca17e899807721af3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260926 13:09:28.744939 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:28.745081 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:28.745113 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d564aeb1a04dcaad076e96a4cd90b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45394
I20260926 13:09:28.745184 8439 tablet_replica.cc:330] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447: stopping tablet replica
I20260926 13:09:28.745225 8439 raft_consensus.cc:2229] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:09:28.745321 8439 raft_consensus.cc:2258] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:28.746383 8439 ts_tablet_manager.cc:1905] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:28.747732 8306 tablet_service.cc:1467] Processing CreateTablet for tablet 2944ac8a36ff4c3b8bfbd430a14de96f (DEFAULT_TABLE table=test-table [id=5beac5f9947a4d23b4a85e4957f4e8cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:28.747851 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2944ac8a36ff4c3b8bfbd430a14de96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.748049 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 2944ac8a36ff4c3b8bfbd430a14de96f (DEFAULT_TABLE table=test-table [id=5beac5f9947a4d23b4a85e4957f4e8cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:28.748136 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2944ac8a36ff4c3b8bfbd430a14de96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.749109 8415 tablet_bootstrap.cc:492] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63: Bootstrap starting.
I20260926 13:09:28.749434 8415 tablet_bootstrap.cc:654] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.749646 8419 tablet_bootstrap.cc:492] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3: Bootstrap starting.
I20260926 13:09:28.749986 8419 tablet_bootstrap.cc:654] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.750569 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 2944ac8a36ff4c3b8bfbd430a14de96f (DEFAULT_TABLE table=test-table [id=5beac5f9947a4d23b4a85e4957f4e8cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:28.750934 8415 tablet_bootstrap.cc:492] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63: No bootstrap required, opened a new log
I20260926 13:09:28.750977 8415 ts_tablet_manager.cc:1397] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 13:09:28.751087 8415 raft_consensus.cc:348] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.751147 8415 raft_consensus.cc:374] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.751163 8415 raft_consensus.cc:729] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc67be09a3984e979ba451801078ef63, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.751205 8415 consensus_queue.cc:260] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.751305 8415 ts_tablet_manager.cc:1428] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.751621 8419 tablet_bootstrap.cc:492] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3: No bootstrap required, opened a new log
I20260926 13:09:28.751669 8419 ts_tablet_manager.cc:1397] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:28.751811 8419 raft_consensus.cc:348] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.751873 8419 raft_consensus.cc:374] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.751889 8419 raft_consensus.cc:729] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427df8ff38bf403ca17e899807721af3, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.751925 8419 consensus_queue.cc:260] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.752005 8419 ts_tablet_manager.cc:1428] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.752404 8439 ts_tablet_manager.cc:1918] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:28.752461 8439 log.cc:1196] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/27d564aeb1a04dcaad076e96a4cd90b3
I20260926 13:09:28.752676 8439 ts_tablet_manager.cc:1939] T 27d564aeb1a04dcaad076e96a4cd90b3 P ce3025b2247d423197b85ea0d89d5447: Deleting consensus metadata
I20260926 13:09:28.752879 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2944ac8a36ff4c3b8bfbd430a14de96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:28.753784 8416 tablet_bootstrap.cc:492] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447: Bootstrap starting.
I20260926 13:09:28.753861 7890 catalog_manager.cc:4841] TS ce3025b2247d423197b85ea0d89d5447 (127.0.98.2:37591): tablet 27d564aeb1a04dcaad076e96a4cd90b3 (table test-table [id=552d76d8eeb5455cb136f426a572ae8f]) successfully deleted
I20260926 13:09:28.754091 8416 tablet_bootstrap.cc:654] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:28.754417 8438 ts_tablet_manager.cc:1905] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:28.755307 8416 tablet_bootstrap.cc:492] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447: No bootstrap required, opened a new log
I20260926 13:09:28.755352 8416 ts_tablet_manager.cc:1397] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:28.755450 8416 raft_consensus.cc:348] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.755513 8416 raft_consensus.cc:374] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:28.755532 8416 raft_consensus.cc:729] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce3025b2247d423197b85ea0d89d5447, State: Initialized, Role: FOLLOWER
I20260926 13:09:28.755570 8416 consensus_queue.cc:260] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.755649 8416 ts_tablet_manager.cc:1428] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:28.756028 8440 ts_tablet_manager.cc:1905] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:28.756070 8438 ts_tablet_manager.cc:1918] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:28.756110 8438 log.cc:1196] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/27d564aeb1a04dcaad076e96a4cd90b3
I20260926 13:09:28.756300 8438 ts_tablet_manager.cc:1939] T 27d564aeb1a04dcaad076e96a4cd90b3 P cc67be09a3984e979ba451801078ef63: Deleting consensus metadata
I20260926 13:09:28.756688 7891 catalog_manager.cc:4841] TS cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109): tablet 27d564aeb1a04dcaad076e96a4cd90b3 (table test-table [id=552d76d8eeb5455cb136f426a572ae8f]) successfully deleted
I20260926 13:09:28.757725 8440 ts_tablet_manager.cc:1918] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:28.757781 8440 log.cc:1196] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-0/wal/wals/27d564aeb1a04dcaad076e96a4cd90b3
I20260926 13:09:28.757952 8440 ts_tablet_manager.cc:1939] T 27d564aeb1a04dcaad076e96a4cd90b3 P 427df8ff38bf403ca17e899807721af3: Deleting consensus metadata
I20260926 13:09:28.758255 7890 catalog_manager.cc:4841] TS 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795): tablet 27d564aeb1a04dcaad076e96a4cd90b3 (table test-table [id=552d76d8eeb5455cb136f426a572ae8f]) successfully deleted
I20260926 13:09:28.849489 8433 raft_consensus.cc:482] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:28.849578 8433 raft_consensus.cc:504] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.849740 8433 leader_election.cc:288] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795), cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109)
I20260926 13:09:28.849942 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2944ac8a36ff4c3b8bfbd430a14de96f" candidate_uuid: "ce3025b2247d423197b85ea0d89d5447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427df8ff38bf403ca17e899807721af3" is_pre_election: true
I20260926 13:09:28.849937 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2944ac8a36ff4c3b8bfbd430a14de96f" candidate_uuid: "ce3025b2247d423197b85ea0d89d5447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc67be09a3984e979ba451801078ef63" is_pre_election: true
I20260926 13:09:28.850024 8061 raft_consensus.cc:2454] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce3025b2247d423197b85ea0d89d5447 in term 0.
I20260926 13:09:28.850032 8341 raft_consensus.cc:2454] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce3025b2247d423197b85ea0d89d5447 in term 0.
I20260926 13:09:28.850162 8134 leader_election.cc:302] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427df8ff38bf403ca17e899807721af3, ce3025b2247d423197b85ea0d89d5447; no voters:
I20260926 13:09:28.850294 8433 raft_consensus.cc:2781] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:28.850337 8433 raft_consensus.cc:482] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:28.850349 8433 raft_consensus.cc:3037] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.851018 8433 raft_consensus.cc:504] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.851135 8433 leader_election.cc:288] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [CANDIDATE]: Term 1 election: Requested vote from peers 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795), cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109)
I20260926 13:09:28.851300 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2944ac8a36ff4c3b8bfbd430a14de96f" candidate_uuid: "ce3025b2247d423197b85ea0d89d5447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "427df8ff38bf403ca17e899807721af3"
I20260926 13:09:28.851365 8061 raft_consensus.cc:3037] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.851337 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2944ac8a36ff4c3b8bfbd430a14de96f" candidate_uuid: "ce3025b2247d423197b85ea0d89d5447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc67be09a3984e979ba451801078ef63"
I20260926 13:09:28.851424 8341 raft_consensus.cc:3037] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:28.852011 8061 raft_consensus.cc:2454] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce3025b2247d423197b85ea0d89d5447 in term 1.
I20260926 13:09:28.852166 8341 raft_consensus.cc:2454] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce3025b2247d423197b85ea0d89d5447 in term 1.
I20260926 13:09:28.852176 8134 leader_election.cc:302] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 427df8ff38bf403ca17e899807721af3, ce3025b2247d423197b85ea0d89d5447; no voters:
I20260926 13:09:28.852257 8433 raft_consensus.cc:2781] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:28.852298 8433 raft_consensus.cc:686] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 1 LEADER]: Becoming Leader. State: Replica: ce3025b2247d423197b85ea0d89d5447, State: Running, Role: LEADER
I20260926 13:09:28.852347 8433 consensus_queue.cc:237] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260926 13:09:28.852969 7902 catalog_manager.cc:5495] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce3025b2247d423197b85ea0d89d5447 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ce3025b2247d423197b85ea0d89d5447" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce3025b2247d423197b85ea0d89d5447" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } health_report { overall_health: UNKNOWN } } }
W20260926 13:09:28.862627 8388 tablet.cc:2369] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:09:28.880524 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.878145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34145 (local address 127.0.98.3:42109)
0926 13:09:28.878197 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.878199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.878292 (+ 93us) server_negotiation.cc:407] Connection header received
0926 13:09:28.878338 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.878339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.878348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.878391 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:28.878488 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.878991 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.879829 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.880019 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.880078 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.880295 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.880324 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.880358 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.880422 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.880456 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 13:09:28.880556 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.878084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42109 (local address 127.0.98.2:34145)
0926 13:09:28.878235 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.878247 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.878299 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.878394 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.878395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.878396 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:28.878454 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.878455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.879056 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.879057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.879735 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.879738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.880144 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.880145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.880204 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.880222 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.880386 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.880388 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.880419 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.880484 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260926 13:09:28.882762 8061 raft_consensus.cc:1261] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Refusing update from remote peer ce3025b2247d423197b85ea0d89d5447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:28.882771 8341 raft_consensus.cc:1261] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Refusing update from remote peer ce3025b2247d423197b85ea0d89d5447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:28.882998 8433 consensus_queue.cc:1035] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc67be09a3984e979ba451801078ef63" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:28.883113 8425 consensus_queue.cc:1035] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "427df8ff38bf403ca17e899807721af3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:28.891233 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.888623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45053 (local address 127.0.98.2:37591)
0926 13:09:28.888673 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:28.888675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:28.888683 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:28.888703 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:28.888704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:28.888713 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:28.888757 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:28.888956 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.889569 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.890416 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.890714 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.890781 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.890999 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.891036 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.891086 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:28.891150 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 13:09:28.891172 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:09:28.891248 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.888068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37591 (local address 127.0.98.3:45053)
0926 13:09:28.888439 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.888465 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:28.888511 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:28.888806 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:28.888807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:28.888808 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:09:28.888863 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.888865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.889659 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.889660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.890246 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:28.890248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:28.890831 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:28.890832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:28.890918 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:28.890923 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:09:28.891095 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:09:28.891097 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:09:28.891147 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 13:09:28.891191 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349}
I20260926 13:09:28.897118 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:28.897332 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:28.898000 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb26d5dde41a4ff59bdf0101372a5abc: Sending DeleteTablet for 3 replicas of tablet 2944ac8a36ff4c3b8bfbd430a14de96f
I20260926 13:09:28.898262 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 2944ac8a36ff4c3b8bfbd430a14de96f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45394
I20260926 13:09:28.898262 8306 tablet_service.cc:1514] Processing DeleteTablet for tablet 2944ac8a36ff4c3b8bfbd430a14de96f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45462
I20260926 13:09:28.898378 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 2944ac8a36ff4c3b8bfbd430a14de96f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36106
I20260926 13:09:28.898401 8438 tablet_replica.cc:330] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63: stopping tablet replica
I20260926 13:09:28.898468 8438 raft_consensus.cc:2229] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:28.898471 8439 tablet_replica.cc:330] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447: stopping tablet replica
I20260926 13:09:28.898521 8438 raft_consensus.cc:2258] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:28.898542 8439 raft_consensus.cc:2229] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:09:28.898610 8439 raft_consensus.cc:2258] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:28.898685 8440 tablet_replica.cc:330] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3: stopping tablet replica
I20260926 13:09:28.898746 8440 raft_consensus.cc:2229] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:28.898780 8440 raft_consensus.cc:2258] T 2944ac8a36ff4c3b8bfbd430a14de96f P 427df8ff38bf403ca17e899807721af3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:28.898885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 7976
I20260926 13:09:28.899070 8438 ts_tablet_manager.cc:1905] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:28.900681 8439 ts_tablet_manager.cc:1905] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:28.900827 8438 ts_tablet_manager.cc:1918] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:28.900887 8438 log.cc:1196] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/2944ac8a36ff4c3b8bfbd430a14de96f
I20260926 13:09:28.901096 8438 ts_tablet_manager.cc:1939] T 2944ac8a36ff4c3b8bfbd430a14de96f P cc67be09a3984e979ba451801078ef63: Deleting consensus metadata
I20260926 13:09:28.901496 7891 catalog_manager.cc:4841] TS cc67be09a3984e979ba451801078ef63 (127.0.98.3:42109): tablet 2944ac8a36ff4c3b8bfbd430a14de96f (table test-table [id=5beac5f9947a4d23b4a85e4957f4e8cb]) successfully deleted
I20260926 13:09:28.902076 8439 ts_tablet_manager.cc:1918] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:09:28.902130 8439 log.cc:1196] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/2944ac8a36ff4c3b8bfbd430a14de96f
I20260926 13:09:28.902408 8439 ts_tablet_manager.cc:1939] T 2944ac8a36ff4c3b8bfbd430a14de96f P ce3025b2247d423197b85ea0d89d5447: Deleting consensus metadata
I20260926 13:09:28.902714 7890 catalog_manager.cc:4841] TS ce3025b2247d423197b85ea0d89d5447 (127.0.98.2:37591): tablet 2944ac8a36ff4c3b8bfbd430a14de96f (table test-table [id=5beac5f9947a4d23b4a85e4957f4e8cb]) successfully deleted
W20260926 13:09:28.905969 7890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32795 (error 108)
I20260926 13:09:28.906068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 8116
I20260926 13:09:28.907440 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:28.907312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32795 (local address 127.0.0.1:36120)
0926 13:09:28.907369 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:28.907408 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 13:09:28.907562 7890 catalog_manager.cc:4584] TS 427df8ff38bf403ca17e899807721af3 (127.0.98.1:32795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2944ac8a36ff4c3b8bfbd430a14de96f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32795: connect: Connection refused (error 111)
I20260926 13:09:28.911724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 8256
I20260926 13:09:28.916878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 7872
2026-09-26T13:09:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2771 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[8505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[8505](info): set up 2 sockets
Sep 26 13:09:28 dist-test-slave-2dw5 krb5kdc[8505](info): commencing operation
krb5kdc: starting...
W20260926 13:09:29.879355 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:29 dist-test-slave-2dw5 krb5kdc[8505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:09:29.999130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36973
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:30.080761 8518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:30.080994 8518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:30.081027 8518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:30.082279 8518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:30.082329 8518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:30.082347 8518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:30.082363 8518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:30.082378 8518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:30.084082 8518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:30.084373 8518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:30.084600 8518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:09:30.089571 8518 server_base.cc:1034] running on GCE node
W20260926 13:09:30.089519 8524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:30.089519 8523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:30.089566 8526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:30.089983 8518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:30.090269 8518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:30.091428 8518 hybrid_clock.cc:648] HybridClock initialized: now 1790428170091403 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T13:09:30Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1180 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:30 dist-test-slave-2dw5 krb5kdc[8531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:30 dist-test-slave-2dw5 krb5kdc[8531](info): set up 2 sockets
Sep 26 13:09:30 dist-test-slave-2dw5 krb5kdc[8531](info): commencing operation
krb5kdc: starting...
W20260926 13:09:31.059247 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:31 dist-test-slave-2dw5 krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:09:31.178299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35531
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:31.258723 8544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:31.258960 8544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:31.258989 8544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:31.260144 8544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:31.260190 8544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:31.260208 8544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:31.260226 8544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:31.260241 8544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:31.261759 8544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:31.262009 8544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:31.262223 8544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:31.267176 8550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:31.267246 8544 server_base.cc:1034] running on GCE node
W20260926 13:09:31.267180 8549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:31.267172 8552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:31.267557 8544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:31.267911 8544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:31.269119 8544 hybrid_clock.cc:648] HybridClock initialized: now 1790428171269103 us; error 28 us; skew 500 ppm
Sep 26 13:09:31 dist-test-slave-2dw5 krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:31.271700 8544 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:09:31.272145 8544 webserver.cc:460] Webserver started at http://127.0.98.62:40937/ using document root <none> and password file <none>
I20260926 13:09:31.272367 8544 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:31.272413 8544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:31.272615 8544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:31.273681 8544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "6b2e4a5302ce4ed197b477cb3b60266d"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "ecf8ac86e213249db6b9931fafaa092b"
server_key_iv: "941a5860f5b39c3a031e6ed7b0bc301b"
server_key_version: "encryptionkey@0"
I20260926 13:09:31.273993 8544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b2e4a5302ce4ed197b477cb3b60266d"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "ecf8ac86e213249db6b9931fafaa092b"
server_key_iv: "941a5860f5b39c3a031e6ed7b0bc301b"
server_key_version: "encryptionkey@0"
I20260926 13:09:31.275288 8544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 13:09:31.276058 8559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:31.276199 8544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:31.276286 8544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "6b2e4a5302ce4ed197b477cb3b60266d"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "ecf8ac86e213249db6b9931fafaa092b"
server_key_iv: "941a5860f5b39c3a031e6ed7b0bc301b"
server_key_version: "encryptionkey@0"
I20260926 13:09:31.276350 8544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:31.285491 8544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:31.287789 8544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:31.287925 8544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:31.292121 8544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35531
I20260926 13:09:31.292156 8611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35531 every 8 connection(s)
I20260926 13:09:31.292629 8544 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:31.292979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 8544
I20260926 13:09:31.293114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:31.293303 392 external_mini_cluster.cc:1428] Setting key c6d286acc8390eb79c93b93585802301
I20260926 13:09:31.293340 8612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:31.296651 8612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d: Bootstrap starting.
I20260926 13:09:31.297292 8612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:31.297547 8612 log.cc:824] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:31.298153 8612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d: No bootstrap required, opened a new log
Sep 26 13:09:31 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:31.300181 8612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35531 } }
I20260926 13:09:31.300328 8612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:31.300362 8612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2e4a5302ce4ed197b477cb3b60266d, State: Initialized, Role: FOLLOWER
I20260926 13:09:31.300487 8612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35531 } }
I20260926 13:09:31.300558 8612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:31.300597 8612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:31.300649 8612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:31.301296 8612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35531 } }
I20260926 13:09:31.301411 8612 leader_election.cc:302] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b2e4a5302ce4ed197b477cb3b60266d; no voters:
I20260926 13:09:31.301607 8612 leader_election.cc:288] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:31.301771 8617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:31.301954 8617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 1 LEADER]: Becoming Leader. State: Replica: 6b2e4a5302ce4ed197b477cb3b60266d, State: Running, Role: LEADER
I20260926 13:09:31.302068 8617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35531 } }
I20260926 13:09:31.302075 8612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:31.302465 8619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b2e4a5302ce4ed197b477cb3b60266d. Latest consensus state: current_term: 1 leader_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35531 } } }
I20260926 13:09:31.302475 8618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35531 } } }
I20260926 13:09:31.302656 8618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:31.302655 8619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:31.302908 8626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:31.303478 8626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:31.304582 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:31.293782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.0.98.62:35531)
0926 13:09:31.293971 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:31.293980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:31.294012 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:09:31.294424 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:31.294433 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:31.294493 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:31.294639 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:31.295295 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.296615 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.297425 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.297933 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.298031 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:31.300365 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:31.300369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:31.300369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:31.300383 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:31.302521 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:31.302992 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:31.302995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:31.302996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:31.303097 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:31.303374 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:31.303375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:31.303375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:31.303678 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:31.304175 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:31.304251 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 13:09:31.304389 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260926 13:09:31.305087 8626 catalog_manager.cc:1347] Generated new cluster ID: 875c3f8046e34503bbee166316672065
I20260926 13:09:31.305137 8626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:31.310439 8626 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:31.311164 8626 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:31.314822 8626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d: Generated new TSK 0
I20260926 13:09:31.315070 8626 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:09:31.343768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35531
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:31.425590 8638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:31.425817 8638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:31.425850 8638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:31.427121 8638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:31.427171 8638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:31.427215 8638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:31.428814 8638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:31.429075 8638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:31.429289 8638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:31.434593 8644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:31.434701 8638 server_base.cc:1034] running on GCE node
W20260926 13:09:31.434578 8646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:31.434578 8643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:31.435170 8638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:31.435446 8638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:31.436643 8638 hybrid_clock.cc:648] HybridClock initialized: now 1790428171436618 us; error 38 us; skew 500 ppm
Sep 26 13:09:31 dist-test-slave-2dw5 krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:31.439366 8638 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:09:31.439848 8638 webserver.cc:460] Webserver started at http://127.0.98.1:40019/ using document root <none> and password file <none>
I20260926 13:09:31.440071 8638 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:31.440122 8638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:31.440255 8638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:31.441287 8638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "d7866b8475634d9189ebd64648b5d22f"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "3f5de1740c4f52b905aa6c4d7d671cd8"
server_key_iv: "2b5c40bfb70e1e025c70346975cc3474"
server_key_version: "encryptionkey@0"
I20260926 13:09:31.441623 8638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d7866b8475634d9189ebd64648b5d22f"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "3f5de1740c4f52b905aa6c4d7d671cd8"
server_key_iv: "2b5c40bfb70e1e025c70346975cc3474"
server_key_version: "encryptionkey@0"
I20260926 13:09:31.442945 8638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:31.443876 8653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:31.444100 8638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:31.444195 8638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "d7866b8475634d9189ebd64648b5d22f"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "3f5de1740c4f52b905aa6c4d7d671cd8"
server_key_iv: "2b5c40bfb70e1e025c70346975cc3474"
server_key_version: "encryptionkey@0"
I20260926 13:09:31.444262 8638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:31.459858 8638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:31.462097 8638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:31.462227 8638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:31.462442 8638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:31.462731 8638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:31.462770 8638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:31.462800 8638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:31.462813 8638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:31.468863 8638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39559
I20260926 13:09:31.468923 8766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39559 every 8 connection(s)
I20260926 13:09:31.469237 8638 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:09:31 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:31.479013 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:31.469998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36377 (local address 127.0.98.62:35531)
0926 13:09:31.470113 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:31.470115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:31.470250 (+ 135us) server_negotiation.cc:407] Connection header received
0926 13:09:31.471261 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:31.471262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:31.471272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:31.471337 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:31.472097 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.472702 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.473740 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.473945 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.473999 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:31.476229 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:31.476230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:31.476231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:31.476239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:31.477687 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:31.478087 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:31.478088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:31.478088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:31.478143 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:31.478367 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:31.478368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:31.478369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:31.478483 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:31.478538 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:31.478806 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 13:09:31.478918 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 13:09:31.479060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 8638
I20260926 13:09:31.479172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:31.479216 8769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:31.469933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:36377)
0926 13:09:31.470125 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:31.470168 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:31.471148 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:31.471408 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:31.471411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:31.471793 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:31.471991 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:31.471996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.472805 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.472806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:31.473596 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:31.473598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.474021 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.474022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:31.474135 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:31.474612 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:31.474620 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:31.476132 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:31.477778 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:31.477786 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:31.477789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:31.478014 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:31.478198 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:31.478198 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:31.478199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:31.478310 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:31.478538 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:31.478553 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:31.478689 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 13:09:31.478778 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:09:31.478984 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20260926 13:09:31.479359 392 external_mini_cluster.cc:1428] Setting key 1577cb5e266578932f804667574d36f2
I20260926 13:09:31.479733 8767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35531
I20260926 13:09:31.479831 8767 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:31.480001 8767 heartbeater.cc:511] Master 127.0.98.62:35531 requested a full tablet report, sending...
I20260926 13:09:31.480515 8575 ts_manager.cc:194] Registered new tserver with Master: d7866b8475634d9189ebd64648b5d22f (127.0.98.1:39559)
I20260926 13:09:31.481316 8575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36377
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:09:31.506278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35531
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:31.588364 8772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:31.588583 8772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:31.588608 8772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:31.589788 8772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:31.589834 8772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:31.589864 8772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:31.591464 8772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:31.591694 8772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:31.591908 8772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:31.596832 8778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:31.596841 8777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:31.596843 8780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:31.597025 8772 server_base.cc:1034] running on GCE node
I20260926 13:09:31.597237 8772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:31.597457 8772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:31.598608 8772 hybrid_clock.cc:648] HybridClock initialized: now 1790428171598593 us; error 29 us; skew 500 ppm
Sep 26 13:09:31 dist-test-slave-2dw5 krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:31.601234 8772 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:09:31.601681 8772 webserver.cc:460] Webserver started at http://127.0.98.2:46771/ using document root <none> and password file <none>
I20260926 13:09:31.601879 8772 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:31.601924 8772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:31.602131 8772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:31.603420 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "c4656f6ab877467caed3a15e77157580"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "ba7bf74aeabd1375b99b88f187d87e2f"
server_key_iv: "7b08698eaec92e0810ca9d7a097a3ad5"
server_key_version: "encryptionkey@0"
I20260926 13:09:31.603833 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c4656f6ab877467caed3a15e77157580"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "ba7bf74aeabd1375b99b88f187d87e2f"
server_key_iv: "7b08698eaec92e0810ca9d7a097a3ad5"
server_key_version: "encryptionkey@0"
I20260926 13:09:31.605123 8772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:31.605837 8787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:31.605984 8772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:31.606066 8772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "c4656f6ab877467caed3a15e77157580"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "ba7bf74aeabd1375b99b88f187d87e2f"
server_key_iv: "7b08698eaec92e0810ca9d7a097a3ad5"
server_key_version: "encryptionkey@0"
I20260926 13:09:31.606127 8772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:31.634500 8772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:31.636684 8772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:31.636828 8772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:31.637055 8772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:31.637357 8772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:31.637389 8772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:31.637419 8772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:31.637436 8772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:31.642959 8772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46781
I20260926 13:09:31.643043 8900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46781 every 8 connection(s)
I20260926 13:09:31.643359 8772 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:09:31 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:31.651492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 8772
I20260926 13:09:31.651585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:31.651878 392 external_mini_cluster.cc:1428] Setting key 9051dd60c097395f93b1a2dbadf25405
I20260926 13:09:31.653810 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:31.644169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43145 (local address 127.0.98.62:35531)
0926 13:09:31.644237 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:31.644240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:31.644534 (+ 294us) server_negotiation.cc:407] Connection header received
0926 13:09:31.645584 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:31.645586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:31.645598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:31.645655 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:31.646368 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.646965 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.648079 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.648317 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.648378 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:31.650825 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:31.650827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:31.650827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:31.650836 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:31.652415 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:31.652865 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:31.652867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:31.652867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:31.652937 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:31.653159 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:31.653160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:31.653161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:31.653299 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:31.653371 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:31.653628 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 13:09:31.653713 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:09:31.654119 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:31.644137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:43145)
0926 13:09:31.644387 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:31.644424 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:31.645447 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:31.645775 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:31.645779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:31.646048 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:31.646272 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:31.646277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.647045 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.647046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:31.647993 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:31.647996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.648390 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.648399 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:31.648515 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:31.649146 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:31.649155 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:31.650678 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:31.652490 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:31.652504 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:31.652507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:31.652781 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:31.653003 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:31.653004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:31.653004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:31.653103 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:31.653368 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:31.653374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:31.653506 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 13:09:31.653594 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 13:09:31.653850 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":112,"threads_started":1}
I20260926 13:09:31.654757 8901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35531
I20260926 13:09:31.654883 8901 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:31.655071 8901 heartbeater.cc:511] Master 127.0.98.62:35531 requested a full tablet report, sending...
I20260926 13:09:31.655445 8575 ts_manager.cc:194] Registered new tserver with Master: c4656f6ab877467caed3a15e77157580 (127.0.98.2:46781)
I20260926 13:09:31.655853 8575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43145
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:09:31.678150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35531
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:31.759723 8906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:31.759953 8906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:31.759979 8906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:31.761188 8906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:31.761236 8906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:31.761264 8906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:31.762887 8906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:31.763113 8906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:31.763330 8906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:31.768321 8912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:31.768412 8911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:31.768442 8906 server_base.cc:1034] running on GCE node
W20260926 13:09:31.768292 8914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:31.768692 8906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:31.768944 8906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:31.770089 8906 hybrid_clock.cc:648] HybridClock initialized: now 1790428171770076 us; error 30 us; skew 500 ppm
Sep 26 13:09:31 dist-test-slave-2dw5 krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:31.772810 8906 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:09:31.773291 8906 webserver.cc:460] Webserver started at http://127.0.98.3:35199/ using document root <none> and password file <none>
I20260926 13:09:31.773469 8906 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:31.773512 8906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:31.773597 8906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:31.774595 8906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "067b1fa22fa74c09b14f44d923496bda"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "a8cd8e9477d7f4b2d3ebfe2d0d3436d8"
server_key_iv: "3036c1572d21323952a97ae8b5a39cdf"
server_key_version: "encryptionkey@0"
I20260926 13:09:31.774921 8906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "067b1fa22fa74c09b14f44d923496bda"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "a8cd8e9477d7f4b2d3ebfe2d0d3436d8"
server_key_iv: "3036c1572d21323952a97ae8b5a39cdf"
server_key_version: "encryptionkey@0"
I20260926 13:09:31.776255 8906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:31.776966 8921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:31.777145 8906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:31.777261 8906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "067b1fa22fa74c09b14f44d923496bda"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "a8cd8e9477d7f4b2d3ebfe2d0d3436d8"
server_key_iv: "3036c1572d21323952a97ae8b5a39cdf"
server_key_version: "encryptionkey@0"
I20260926 13:09:31.777320 8906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:31.801222 8906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:31.803460 8906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:31.803598 8906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:31.803825 8906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:31.804116 8906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:31.804149 8906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:31.804177 8906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:31.804194 8906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:31.809664 8906 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43103
I20260926 13:09:31.809731 9034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43103 every 8 connection(s)
I20260926 13:09:31.810065 8906 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:31.813104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 8906
I20260926 13:09:31.813186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:31.813369 392 external_mini_cluster.cc:1428] Setting key 82e7a4be5dfdde98f9c1d407271e1cf2
Sep 26 13:09:31 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:31.819283 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:31.810777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53313 (local address 127.0.98.62:35531)
0926 13:09:31.810825 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:31.810827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:31.811021 (+ 194us) server_negotiation.cc:407] Connection header received
0926 13:09:31.811946 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:31.811947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:31.811958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:31.812049 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:31.812661 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.813199 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.814336 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.814525 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.814575 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:31.816865 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:31.816866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:31.816866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:31.816873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:31.818098 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:31.818439 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:31.818440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:31.818440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:31.818484 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:31.818663 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:31.818663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:31.818664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:31.818876 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:31.818932 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:31.819155 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 13:09:31.819213 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:09:31.819473 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:31.810737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:53313)
0926 13:09:31.810908 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:31.810943 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:31.811839 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:31.812101 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:31.812104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:31.812352 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:31.812537 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:31.812541 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.813300 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.813301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:31.814268 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:31.814271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:31.814709 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:31.814710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:31.814903 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:31.815362 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:31.815378 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:31.816771 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:31.818172 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:31.818177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:31.818179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:31.818381 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:31.818530 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:31.818530 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:31.818530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:31.818623 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:31.818929 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:31.818934 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:31.819052 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 13:09:31.819144 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:09:31.819266 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260926 13:09:31.819887 9035 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35531
I20260926 13:09:31.819979 9035 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:31.820142 9035 heartbeater.cc:511] Master 127.0.98.62:35531 requested a full tablet report, sending...
I20260926 13:09:31.820461 8575 ts_manager.cc:194] Registered new tserver with Master: 067b1fa22fa74c09b14f44d923496bda (127.0.98.3:43103)
I20260926 13:09:31.820798 8575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53313
I20260926 13:09:31.824918 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 13:09:31.902428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 8638
I20260926 13:09:31.907317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 8772
I20260926 13:09:31.911749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 8906
I20260926 13:09:31.915729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 8544
I20260926 13:09:31.920047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35531
--webserver_interface=127.0.98.62
--webserver_port=40937
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:32.002063 9044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:32.002301 9044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:32.002331 9044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:32.003615 9044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:32.003665 9044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:32.003683 9044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:32.003700 9044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:32.003731 9044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:32.005321 9044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:32.005590 9044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:32.005841 9044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:32.010864 9052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:32.010895 9050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:32.010864 9049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:32.011029 9044 server_base.cc:1034] running on GCE node
I20260926 13:09:32.011327 9044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:32.011632 9044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:32.012784 9044 hybrid_clock.cc:648] HybridClock initialized: now 1790428172012755 us; error 44 us; skew 500 ppm
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:32.015439 9044 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:09:32.015913 9044 webserver.cc:460] Webserver started at http://127.0.98.62:40937/ using document root <none> and password file <none>
I20260926 13:09:32.016110 9044 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:32.016172 9044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:32.017505 9044 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:32.018220 9059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:32.018369 9044 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:32.018453 9044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "6b2e4a5302ce4ed197b477cb3b60266d"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "ecf8ac86e213249db6b9931fafaa092b"
server_key_iv: "941a5860f5b39c3a031e6ed7b0bc301b"
server_key_version: "encryptionkey@0"
I20260926 13:09:32.018821 9044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:32.028638 9044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:32.030977 9044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:32.031128 9044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:32.035151 9044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35531
I20260926 13:09:32.035192 9111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35531 every 8 connection(s)
I20260926 13:09:32.035759 9044 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:32.037070 9112 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:09:32.037693 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d: Bootstrap starting.
I20260926 13:09:32.044999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 9044
I20260926 13:09:32.045295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39559
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35531
--webserver_port=40019
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:09:32.045670 9112 log.cc:824] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:32.047793 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:09:32.048053 9112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d: Bootstrap complete.
I20260926 13:09:32.050236 9112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35531 } }
I20260926 13:09:32.050473 9112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2e4a5302ce4ed197b477cb3b60266d, State: Initialized, Role: FOLLOWER
I20260926 13:09:32.050616 9112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35531 } }
I20260926 13:09:32.050680 9112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:32.050726 9112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:32.050770 9112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:09:32.051831 9112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35531 } }
I20260926 13:09:32.051925 9112 leader_election.cc:302] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b2e4a5302ce4ed197b477cb3b60266d; no voters:
I20260926 13:09:32.052090 9112 leader_election.cc:288] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:09:32.052140 9116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:09:32.052278 9116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [term 2 LEADER]: Becoming Leader. State: Replica: 6b2e4a5302ce4ed197b477cb3b60266d, State: Running, Role: LEADER
I20260926 13:09:32.052380 9112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:32.052423 9116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35531 } }
I20260926 13:09:32.052793 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35531 } } }
I20260926 13:09:32.052897 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:32.052788 9116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b2e4a5302ce4ed197b477cb3b60266d. Latest consensus state: current_term: 2 leader_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b2e4a5302ce4ed197b477cb3b60266d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35531 } } }
I20260926 13:09:32.053066 9116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:32.053166 9124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:32.053669 9124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:32.053791 9124 catalog_manager.cc:1259] Loaded cluster ID: 875c3f8046e34503bbee166316672065
I20260926 13:09:32.053969 9124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:32.055241 9124 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:32.055428 9124 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b2e4a5302ce4ed197b477cb3b60266d: Loaded TSK: 0
I20260926 13:09:32.055639 9124 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:09:32.129426 9115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:32.129635 9115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:32.129667 9115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:32.130955 9115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:32.131003 9115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:32.131031 9115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:32.132548 9115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40019
--tserver_master_addrs=127.0.98.62:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:32.132807 9115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:32.133023 9115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:32.139091 9139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:32.139118 9115 server_base.cc:1034] running on GCE node
W20260926 13:09:32.139091 9141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:32.139266 9138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:32.139528 9115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:32.139950 9115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:32.141120 9115 hybrid_clock.cc:648] HybridClock initialized: now 1790428172141105 us; error 55 us; skew 500 ppm
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428172, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:32.143801 9115 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 13:09:32.144286 9115 webserver.cc:460] Webserver started at http://127.0.98.1:40019/ using document root <none> and password file <none>
I20260926 13:09:32.144639 9115 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:32.144712 9115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:32.146092 9115 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:09:32.146654 9148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:32.146818 9115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:09:32.146929 9115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "d7866b8475634d9189ebd64648b5d22f"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "3f5de1740c4f52b905aa6c4d7d671cd8"
server_key_iv: "2b5c40bfb70e1e025c70346975cc3474"
server_key_version: "encryptionkey@0"
I20260926 13:09:32.147286 9115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:32.167779 9115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:32.169960 9115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:32.170100 9115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:32.170324 9115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:32.170663 9115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:32.170701 9115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:32.170743 9115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:32.170766 9115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:32.177425 9115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39559
I20260926 13:09:32.177495 9261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39559 every 8 connection(s)
I20260926 13:09:32.177850 9115 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:09:32.180857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 9115
I20260926 13:09:32.181151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46781
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35531
--webserver_port=46771
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:32.188526 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:32.178552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:43239)
0926 13:09:32.178740 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:32.178791 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:32.179393 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:32.179803 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:32.179807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:32.180532 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:32.180829 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.180836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.182479 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.182481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.183780 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.183783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.184735 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.184737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.184932 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.185835 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:32.185846 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:32.188283 (+ 2437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
W20260926 13:09:32.188586 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:32.178652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43239 (local address 127.0.98.62:35531)
0926 13:09:32.178827 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:32.178854 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:32.178894 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:09:32.179575 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:32.179580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:32.179630 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:32.179749 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:32.180970 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.182342 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.183901 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.184614 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.184739 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.188445 (+ 3706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43239: BlockingRecv error: recv got EOF from 127.0.98.1:43239 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
W20260926 13:09:32.188781 9262 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35531 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35531: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:32.194105 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:32.189016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:41739)
0926 13:09:32.189082 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:32.189095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:32.189164 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:32.189367 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:32.189368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:32.189527 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:32.189590 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.189592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.190374 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.190375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.191223 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.191226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.191571 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.191572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.191641 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.192063 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:32.192069 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:32.194017 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 13:09:32.194172 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:32.189058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41739 (local address 127.0.98.62:35531)
0926 13:09:32.189109 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:32.189111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:32.189144 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:09:32.189269 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:32.189270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:32.189286 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:32.189343 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:32.189680 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.190290 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.191309 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.191498 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.191554 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.194119 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41739: BlockingRecv error: recv got EOF from 127.0.98.1:41739 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:32.198732 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:32.194455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:56343)
0926 13:09:32.194514 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:32.194523 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:32.194559 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:32.194693 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:32.194693 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:32.194807 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:32.194857 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.194859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.195492 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.195493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.196055 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.196058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.196444 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.196445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.196494 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.196938 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:32.196943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:32.198648 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 13:09:32.198913 9262 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:09:32.199039 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:32.194524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56343 (local address 127.0.98.62:35531)
0926 13:09:32.194579 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:32.194581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:32.194589 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:32.194612 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:32.194613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:32.194624 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:32.194688 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:32.194915 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.195424 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.196197 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.196378 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.196440 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.198979 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56343: BlockingRecv error: recv got EOF from 127.0.98.1:56343 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 13:09:32.264665 9266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:32.264887 9266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:32.264910 9266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:32.266127 9266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:32.266172 9266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:32.266198 9266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:32.267686 9266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46771
--tserver_master_addrs=127.0.98.62:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:32.267925 9266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:32.268150 9266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:32.273216 9271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:32.273211 9272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:32.273240 9274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:32.273248 9266 server_base.cc:1034] running on GCE node
I20260926 13:09:32.273523 9266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:32.273761 9266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:32.274922 9266 hybrid_clock.cc:648] HybridClock initialized: now 1790428172274907 us; error 28 us; skew 500 ppm
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428172, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:32.277453 9266 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 13:09:32.277925 9266 webserver.cc:460] Webserver started at http://127.0.98.2:46771/ using document root <none> and password file <none>
I20260926 13:09:32.278158 9266 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:32.278219 9266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:32.279681 9266 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:09:32.280298 9281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:32.280457 9266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:32.280540 9266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "c4656f6ab877467caed3a15e77157580"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "ba7bf74aeabd1375b99b88f187d87e2f"
server_key_iv: "7b08698eaec92e0810ca9d7a097a3ad5"
server_key_version: "encryptionkey@0"
I20260926 13:09:32.280925 9266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:32.323751 9266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:32.325812 9266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:32.325949 9266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:32.326172 9266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:32.326449 9266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:32.326483 9266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:32.326511 9266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:32.326532 9266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:32.332136 9266 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46781
I20260926 13:09:32.332204 9394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46781 every 8 connection(s)
I20260926 13:09:32.332520 9266 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:09:32.337357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 9266
I20260926 13:09:32.337616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43103
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35531
--webserver_port=35199
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:32.339968 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:32.333198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:56921)
0926 13:09:32.333341 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:32.333383 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:32.333864 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:32.334093 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:32.334098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:32.334642 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:32.334836 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.334869 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.335553 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.335554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.336361 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.336363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.336722 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.336723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.336847 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.337394 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:32.337403 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:32.339781 (+ 2378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
W20260926 13:09:32.339967 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:32.333233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56921 (local address 127.0.98.62:35531)
0926 13:09:32.333282 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:32.333283 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:32.333462 (+ 179us) server_negotiation.cc:407] Connection header received
0926 13:09:32.333960 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:32.333961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:32.333977 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:32.334025 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:32.334957 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.335475 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.336446 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.336655 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.336707 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.339906 (+ 3199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56921: BlockingRecv error: recv got EOF from 127.0.98.2:56921 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 13:09:32.340284 9395 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35531 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35531: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:32.345098 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:32.340499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:48293)
0926 13:09:32.340602 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:32.340614 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:32.340666 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:32.340782 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:32.340782 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:32.340926 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:32.340981 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.340983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.341582 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.341582 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.342353 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.342355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.342678 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.342679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.342743 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.343186 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:32.343191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:32.345025 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86}
W20260926 13:09:32.345175 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:32.340554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48293 (local address 127.0.98.62:35531)
0926 13:09:32.340596 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:32.340598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:32.340652 (+ 54us) server_negotiation.cc:407] Connection header received
0926 13:09:32.340714 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:32.340715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:32.340723 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:32.340768 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:32.341052 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.341513 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.342436 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.342613 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.342664 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.345123 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48293: BlockingRecv error: recv got EOF from 127.0.98.2:48293 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:32.349478 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:32.345426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:56239)
0926 13:09:32.345468 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:32.345477 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:32.345511 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:32.345619 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:32.345620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:32.345735 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:32.345763 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.345764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.346351 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.346351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.346828 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.346829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.347172 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.347173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.347227 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.347685 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:32.347690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:32.349402 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 13:09:32.349546 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:32.345483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56239 (local address 127.0.98.62:35531)
0926 13:09:32.345514 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:32.345516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:32.345523 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:09:32.345564 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:32.345565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:32.345574 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:32.345617 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:32.345825 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.346284 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.346922 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.347109 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.347159 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.349488 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56239: BlockingRecv error: recv got EOF from 127.0.98.2:56239 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 13:09:32.349686 9395 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:09:32.419255 9398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:32.419476 9398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:32.419507 9398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:32.420750 9398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:32.420804 9398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:32.420841 9398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:32.422394 9398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35199
--tserver_master_addrs=127.0.98.62:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:32.422646 9398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:32.422928 9398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:32.428143 9406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:32.428179 9403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:32.428140 9404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:32.428180 9398 server_base.cc:1034] running on GCE node
I20260926 13:09:32.428546 9398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:32.428881 9398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:32.430038 9398 hybrid_clock.cc:648] HybridClock initialized: now 1790428172430027 us; error 28 us; skew 500 ppm
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428172, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:32.432660 9398 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 13:09:32.433173 9398 webserver.cc:460] Webserver started at http://127.0.98.3:35199/ using document root <none> and password file <none>
I20260926 13:09:32.433390 9398 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:32.433457 9398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:32.435076 9398 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:32.435854 9413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:32.436188 9398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:32.436291 9398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "067b1fa22fa74c09b14f44d923496bda"
format_stamp: "Formatted at 2026-09-26 13:09:31 on dist-test-slave-2dw5"
server_key: "a8cd8e9477d7f4b2d3ebfe2d0d3436d8"
server_key_iv: "3036c1572d21323952a97ae8b5a39cdf"
server_key_version: "encryptionkey@0"
I20260926 13:09:32.436796 9398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:32.461135 9398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:32.463618 9398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:32.463762 9398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:32.463990 9398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:32.464316 9398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:32.464356 9398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:32.464392 9398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:32.464416 9398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:32.470919 9398 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43103
I20260926 13:09:32.470968 9526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43103 every 8 connection(s)
I20260926 13:09:32.471334 9398 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:32.473047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 9398
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:09:32.480387 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:32.472088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59113 (local address 127.0.98.62:35531)
0926 13:09:32.472158 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:32.472160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:32.472379 (+ 219us) server_negotiation.cc:407] Connection header received
0926 13:09:32.472991 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:32.472993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:32.473006 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:32.473049 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:32.474423 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.475200 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.476652 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.476923 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.476973 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.480336 (+ 3363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59113: BlockingRecv error: recv got EOF from 127.0.98.3:59113 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:09:32.480541 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:32.472065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:59113)
0926 13:09:32.472228 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:32.472271 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:32.472841 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:32.473261 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:32.473265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:32.474004 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:32.474298 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.474306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.475289 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.475290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.476574 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.476576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.477009 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.477010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.477120 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.477470 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:32.477479 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:32.480263 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
W20260926 13:09:32.480786 9527 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35531 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35531: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260926 13:09:32.481606 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:32.473632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.0.98.62:35531)
0926 13:09:32.473833 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:32.473834 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:32.473842 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:32.473862 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:32.473863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:32.473871 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:32.473926 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:32.474669 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.475360 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.476438 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.476667 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.476731 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.478105 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:32.478109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:32.478109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:32.478120 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:32.480110 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:32.480538 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:32.480542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:32.480542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:32.480663 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:32.480857 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:32.480857 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:32.480858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:32.481137 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:32.481287 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:32.481397 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 13:09:32.481510 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":49,"threads_started":1}
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:32.486012 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:32.481004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:44069)
0926 13:09:32.481104 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:32.481116 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:32.481165 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:32.481280 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:32.481281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:32.481434 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:32.481496 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.481497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.482422 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.482423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.483058 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.483060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.483419 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.483420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.483480 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.484018 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:32.484023 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:32.485941 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260926 13:09:32.486094 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:32.481057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44069 (local address 127.0.98.62:35531)
0926 13:09:32.481107 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:32.481109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:32.481160 (+ 51us) server_negotiation.cc:407] Connection header received
0926 13:09:32.481212 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:32.481212 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:32.481223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:32.481276 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:32.481607 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.482364 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.483126 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.483333 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.483394 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.486042 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44069: BlockingRecv error: recv got EOF from 127.0.98.3:44069 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:32 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:32.490890 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:32.486489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:41467)
0926 13:09:32.486584 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:32.486594 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:32.486625 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:32.486830 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:32.486831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:32.487015 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:32.487050 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.487052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.487758 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.487759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.488273 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:32.488275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.488629 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.488630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:32.488670 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.489132 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:32.489136 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:32.490800 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260926 13:09:32.491039 9265 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:32.486572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41467 (local address 127.0.98.62:35531)
0926 13:09:32.486618 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:32.486619 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:32.486629 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:32.486681 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:32.486681 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:32.486697 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:32.486736 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:32.487122 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.487595 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.488359 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:32.488539 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:32.488597 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:32.490979 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41467: BlockingRecv error: recv got EOF from 127.0.98.3:41467 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 13:09:32.491104 9527 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 13:09:33 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:33 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:33.203822 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:33.199283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:54887)
0926 13:09:33.199481 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:33.199493 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:33.199558 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:33.199688 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:33.199690 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:33.199855 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:33.199903 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:33.199905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:33.200556 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:33.200557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:33.201077 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:33.201079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:33.201417 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:33.201418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:33.201467 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:33.201886 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:33.201891 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:33.203727 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
W20260926 13:09:33.203976 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:33.199354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54887 (local address 127.0.98.62:35531)
0926 13:09:33.199475 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:33.199478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:33.199556 (+ 78us) server_negotiation.cc:407] Connection header received
0926 13:09:33.199618 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:33.199618 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:33.199630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:33.199685 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:33.199992 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:33.200484 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:33.201148 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:33.201350 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:33.201403 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:33.203833 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54887: BlockingRecv error: recv got EOF from 127.0.98.1:54887 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
Sep 26 13:09:33 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:33 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:33.355095 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:33.350122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:60759)
0926 13:09:33.350259 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:33.350288 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:33.350359 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:33.350553 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:33.350554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:33.350780 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:33.350860 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:33.350863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:33.351699 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:33.351700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:33.352337 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:33.352339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:33.352694 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:33.352695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:33.352744 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:33.353209 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:33.353214 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:33.355003 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
W20260926 13:09:33.355145 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:33.350235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60759 (local address 127.0.98.62:35531)
0926 13:09:33.350290 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:33.350291 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:33.350350 (+ 59us) server_negotiation.cc:407] Connection header received
0926 13:09:33.350428 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:33.350429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:33.350440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:33.350490 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:33.350966 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:33.351614 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:33.352402 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:33.352613 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:33.352666 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:33.355092 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60759: BlockingRecv error: recv got EOF from 127.0.98.2:60759 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 13:09:33 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:33 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:33.496191 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:33.491526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:50255)
0926 13:09:33.491658 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:33.491672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:33.491745 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:33.491968 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:33.491969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:33.492124 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:33.492169 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:33.492170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:33.492860 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:33.492861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:33.493418 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:33.493421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:33.493745 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:33.493746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:33.493796 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:33.494240 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:33.494245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:33.496123 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
W20260926 13:09:33.496306 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:33.491626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50255 (local address 127.0.98.62:35531)
0926 13:09:33.491718 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:33.491720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:33.491730 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:33.491811 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:33.491813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:33.491825 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:33.491873 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:33.492262 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:33.492780 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:33.493501 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:33.493685 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:33.493739 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:33.496234 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50255: BlockingRecv error: recv got EOF from 127.0.98.3:50255 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 13:09:34 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:34 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:34.214998 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:34.204420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:48357)
0926 13:09:34.204616 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:34.204636 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:34.204714 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:34.204881 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:34.204882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:34.205119 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:34.205172 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:34.205174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:34.205998 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:34.205999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:34.206619 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:34.206621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:34.207101 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:34.207102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:34.207147 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:34.212840 (+ 5693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:34.212846 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:34.214920 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":99,"threads_started":1}
W20260926 13:09:34.215313 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:34.204577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48357 (local address 127.0.98.62:35531)
0926 13:09:34.204707 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:34.204709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:34.204720 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:34.204778 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:34.204779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:34.204792 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:34.204861 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:34.205264 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:34.205912 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:34.206725 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:34.206980 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:34.207047 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:34.215222 (+ 8175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48357: BlockingRecv error: recv got EOF from 127.0.98.1:48357 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":54,"threads_started":1}
Sep 26 13:09:34 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:34 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:34.360728 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:34.355616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:32971)
0926 13:09:34.355765 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:34.355781 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:34.355871 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:34.356018 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:34.356019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:34.356208 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:34.356284 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:34.356286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:34.356974 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:34.356975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:34.357583 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:34.357584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:34.357956 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:34.357957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:34.358016 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:34.358588 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:34.358593 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:34.360655 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
W20260926 13:09:34.360888 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:34.355708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32971 (local address 127.0.98.62:35531)
0926 13:09:34.355767 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:34.355769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:34.355862 (+ 93us) server_negotiation.cc:407] Connection header received
0926 13:09:34.355939 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:34.355940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:34.355952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:34.355999 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:34.356352 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:34.356895 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:34.357660 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:34.357858 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:34.357917 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:34.360799 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32971: BlockingRecv error: recv got EOF from 127.0.98.2:32971 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 26 13:09:34 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:34 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:34.502074 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:34.496857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:52513)
0926 13:09:34.497074 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:34.497088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:34.497189 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:34.497445 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:34.497446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:34.497621 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:34.497673 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:34.497674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:34.498442 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:34.498443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:34.499156 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:34.499158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:34.499547 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:34.499548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:34.499608 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:34.500208 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:34.500215 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:34.502003 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":119,"threads_started":1}
W20260926 13:09:34.502255 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:34.497031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52513 (local address 127.0.98.62:35531)
0926 13:09:34.497102 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:34.497104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:34.497168 (+ 64us) server_negotiation.cc:407] Connection header received
0926 13:09:34.497297 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:34.497299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:34.497316 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:34.497379 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:34.497754 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:34.498370 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:34.499229 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:34.499432 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:34.499491 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:34.502163 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52513: BlockingRecv error: recv got EOF from 127.0.98.3:52513 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 13:09:35 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:35 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:35.220450 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:35.215670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:40773)
0926 13:09:35.215808 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:35.215819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:35.215867 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:35.216005 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:35.216005 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:35.216260 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:35.216310 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:35.216312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:35.217018 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:35.217019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:35.217561 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:35.217562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:35.217925 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:35.217926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:35.217975 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:35.218534 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:35.218539 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:35.220361 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
W20260926 13:09:35.220523 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:35.215763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40773 (local address 127.0.98.62:35531)
0926 13:09:35.215877 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:35.215879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:35.215886 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:09:35.215914 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:35.215914 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:35.215924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:35.215992 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:35.216383 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:35.216931 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:35.217635 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:35.217838 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:35.217889 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:35.220461 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40773: BlockingRecv error: recv got EOF from 127.0.98.1:40773 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
Sep 26 13:09:35 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:35 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:35.366194 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:35.361309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:43945)
0926 13:09:35.361445 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:35.361459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:35.361539 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:35.361687 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:35.361688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:35.361856 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:35.361907 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:35.361910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:35.362565 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:35.362566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:35.363266 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:35.363269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:35.363635 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:35.363636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:35.363709 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:35.364230 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:35.364235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:35.366104 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
W20260926 13:09:35.366369 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:35.361389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43945 (local address 127.0.98.62:35531)
0926 13:09:35.361475 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:35.361477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:35.361517 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:09:35.361605 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:35.361606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:35.361618 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:35.361666 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:35.361981 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:35.362490 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:35.363361 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:35.363567 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:35.363628 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:35.366314 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43945: BlockingRecv error: recv got EOF from 127.0.98.2:43945 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 13:09:35 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:35 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:35.507417 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:35.502556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:38679)
0926 13:09:35.502730 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:35.502740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:35.502794 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:35.503004 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:35.503005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:35.503214 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:35.503267 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:35.503269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:35.503965 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:35.503966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:35.504521 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:35.504524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:35.504932 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:35.504933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:35.504977 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:35.505463 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:35.505468 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:35.507357 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
W20260926 13:09:35.507530 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:35.502632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38679 (local address 127.0.98.62:35531)
0926 13:09:35.502696 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:35.502699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:35.502781 (+ 82us) server_negotiation.cc:407] Connection header received
0926 13:09:35.502866 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:35.502867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:35.502878 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:35.502918 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:35.503377 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:35.503892 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:35.504672 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:35.504859 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:35.504920 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:35.507459 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38679: BlockingRecv error: recv got EOF from 127.0.98.3:38679 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 13:09:36 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:36 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:36.226609 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:36.221112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:46949)
0926 13:09:36.221299 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:36.221331 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:36.221409 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:36.221612 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:36.221613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:36.221814 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:36.221885 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:36.221887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:36.222691 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:36.222692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:36.223427 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:36.223430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:36.223897 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:36.223898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:36.223959 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:36.224626 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:36.224631 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:36.226507 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
W20260926 13:09:36.226692 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:36.221222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46949 (local address 127.0.98.62:35531)
0926 13:09:36.221399 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:36.221401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:36.221415 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:09:36.221473 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:36.221473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:36.221484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:36.221595 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:36.221975 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:36.222593 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:36.223506 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:36.223802 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:36.223878 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:36.226626 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46949: BlockingRecv error: recv got EOF from 127.0.98.1:46949 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":77,"threads_started":1}
Sep 26 13:09:36 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:36 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:36.371596 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:36.366899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:34083)
0926 13:09:36.367108 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:36.367125 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:36.367222 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:36.367428 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:36.367428 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:36.367603 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:36.367649 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:36.367651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:36.368351 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:36.368352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:36.368950 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:36.368952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:36.369307 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:36.369308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:36.369360 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:36.369870 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:36.369876 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:36.371518 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
W20260926 13:09:36.371675 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:36.366983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34083 (local address 127.0.98.62:35531)
0926 13:09:36.367114 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:36.367116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:36.367211 (+ 95us) server_negotiation.cc:407] Connection header received
0926 13:09:36.367351 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:36.367352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:36.367365 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:36.367411 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:36.367723 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:36.368261 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:36.369024 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:36.369241 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:36.369300 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:36.371618 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34083: BlockingRecv error: recv got EOF from 127.0.98.2:34083 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 26 13:09:36 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:36 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:36.512995 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:36.507939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:42565)
0926 13:09:36.508183 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:36.508198 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:36.508285 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:36.508455 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:36.508456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:36.508657 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:36.508727 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:36.508729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:36.509358 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:36.509359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:36.509958 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:36.509960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:36.510263 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:36.510264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:36.510305 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:36.510785 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:36.510790 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:36.512907 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":93,"threads_started":1}
W20260926 13:09:36.513079 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:36.508079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42565 (local address 127.0.98.62:35531)
0926 13:09:36.508128 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:36.508129 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:36.508332 (+ 203us) server_negotiation.cc:407] Connection header received
0926 13:09:36.508359 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:36.508360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:36.508373 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:36.508430 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:36.508792 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:36.509290 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:36.510025 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:36.510207 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:36.510312 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:36.513020 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42565: BlockingRecv error: recv got EOF from 127.0.98.3:42565 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 13:09:37 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:37 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:37.232190 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:37.227162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:42017)
0926 13:09:37.227298 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:37.227308 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:37.227361 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:37.227612 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:37.227614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:37.227808 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:37.227866 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:37.227868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:37.228652 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:37.228653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:37.229359 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:37.229362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:37.229738 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:37.229739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:37.229790 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:37.230243 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:37.230248 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:37.232110 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
W20260926 13:09:37.232277 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:37.227279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42017 (local address 127.0.98.62:35531)
0926 13:09:37.227433 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:37.227435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:37.227444 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:37.227471 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:37.227471 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:37.227486 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:37.227560 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:37.227938 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:37.228546 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:37.229439 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:37.229645 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:37.229712 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:37.232219 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42017: BlockingRecv error: recv got EOF from 127.0.98.1:42017 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
Sep 26 13:09:37 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:37 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:37.376217 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:37.372106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:57675)
0926 13:09:37.372227 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:37.372237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:37.372283 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:37.372385 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:37.372385 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:37.372538 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:37.372585 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:37.372586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:37.373184 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:37.373185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:37.373715 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:37.373717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:37.374039 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:37.374039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:37.374075 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:37.374488 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:37.374492 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:37.376158 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
W20260926 13:09:37.376305 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:37.372223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57675 (local address 127.0.98.62:35531)
0926 13:09:37.372267 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:37.372268 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:37.372276 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:09:37.372324 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:37.372325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:37.372334 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:37.372373 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:37.372633 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:37.373121 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:37.373785 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:37.373986 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:37.374036 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:37.376247 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57675: BlockingRecv error: recv got EOF from 127.0.98.2:57675 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 13:09:37 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:37 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:37.518390 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:37.513633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:58967)
0926 13:09:37.513820 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:37.513832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:37.513908 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:37.514030 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:37.514031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:37.514192 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:37.514248 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:37.514250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:37.515010 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:37.515012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:37.515701 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:37.515704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:37.516033 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:37.516033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:37.516085 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:37.516566 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:37.516569 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:37.518300 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
W20260926 13:09:37.518580 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:37.513751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58967 (local address 127.0.98.62:35531)
0926 13:09:37.513824 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:37.513825 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:37.513899 (+ 74us) server_negotiation.cc:407] Connection header received
0926 13:09:37.513964 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:37.513965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:37.513975 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:37.514013 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:37.514303 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:37.514899 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:37.515771 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:37.515967 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:37.516020 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:37.518523 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58967: BlockingRecv error: recv got EOF from 127.0.98.3:58967 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 13:09:38 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:38 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:38.238140 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:38.232797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:60859)
0926 13:09:38.232948 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:38.232962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:38.233033 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:38.233200 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:38.233201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:38.233387 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:38.233452 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:38.233454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:38.234125 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:38.234126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:38.234722 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:38.234724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:38.235230 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:38.235232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:38.235300 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:38.235832 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:38.235837 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:38.238056 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
W20260926 13:09:38.238276 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:38.232885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60859 (local address 127.0.98.62:35531)
0926 13:09:38.233037 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:38.233039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:38.233081 (+ 42us) server_negotiation.cc:407] Connection header received
0926 13:09:38.233105 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:38.233105 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:38.233118 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:38.233186 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:38.233523 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:38.234052 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:38.234806 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:38.235096 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:38.235156 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:38.238190 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60859: BlockingRecv error: recv got EOF from 127.0.98.1:60859 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
Sep 26 13:09:38 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:38 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:38.383111 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:38.376843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:46903)
0926 13:09:38.377045 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:38.377055 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:38.377137 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:38.377380 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:38.377380 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:38.377554 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:38.377617 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:38.377619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:38.378617 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:38.378618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:38.379511 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:38.379515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:38.379861 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:38.379862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:38.379911 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:38.380395 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:38.380401 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:38.382995 (+ 2594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":60,"threads_started":1}
W20260926 13:09:38.383190 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:38.376967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46903 (local address 127.0.98.62:35531)
0926 13:09:38.377059 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:38.377062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:38.377118 (+ 56us) server_negotiation.cc:407] Connection header received
0926 13:09:38.377224 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:38.377225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:38.377245 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:38.377352 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:38.377730 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:38.378514 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:38.379596 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:38.379792 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:38.379864 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:38.383115 (+ 3251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46903: BlockingRecv error: recv got EOF from 127.0.98.2:46903 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 13:09:38 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:38 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:38.524834 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:38.519066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:37389)
0926 13:09:38.519243 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:38.519257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:38.519331 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:38.519630 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:38.519631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:38.519839 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:38.519908 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:38.519910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:38.520695 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:38.520696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:38.521377 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:38.521380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:38.521816 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:38.521816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:38.521867 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:38.522512 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:38.522517 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:38.524718 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
W20260926 13:09:38.524937 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:38.519180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37389 (local address 127.0.98.62:35531)
0926 13:09:38.519271 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:38.519273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:38.519320 (+ 47us) server_negotiation.cc:407] Connection header received
0926 13:09:38.519460 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:38.519460 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:38.519476 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:38.519563 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:38.520017 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:38.520596 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:38.521511 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:38.521710 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:38.521814 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:38.524855 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37389: BlockingRecv error: recv got EOF from 127.0.98.3:37389 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 13:09:39 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:39 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:39.254582 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:39.238748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:56765)
0926 13:09:39.238916 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:39.238927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:39.238978 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:39.239231 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:39.239232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:39.239433 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:39.239491 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:39.239493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:39.240188 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:39.240189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:39.240721 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:39.240722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:39.241137 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:39.241138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:39.241190 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:39.252449 (+ 11259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:39.252456 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:39.254499 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
W20260926 13:09:39.254750 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:39.238923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56765 (local address 127.0.98.62:35531)
0926 13:09:39.239075 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:39.239077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:39.239087 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:39.239112 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:39.239112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:39.239122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:39.239191 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:39.239566 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:39.240108 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:39.240807 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:39.241011 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:39.241073 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:39.254687 (+ 13614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56765: BlockingRecv error: recv got EOF from 127.0.98.1:56765 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
Sep 26 13:09:39 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:39 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:39.388695 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:39.383880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:55111)
0926 13:09:39.383996 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:39.384007 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:39.384061 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:39.384200 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:39.384200 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:39.384349 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:39.384398 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:39.384400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:39.385084 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:39.385085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:39.385841 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:39.385843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:39.386160 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:39.386160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:39.386194 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:39.386643 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:39.386648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:39.388574 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
W20260926 13:09:39.388756 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:39.383998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55111 (local address 127.0.98.62:35531)
0926 13:09:39.384050 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:39.384051 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:39.384066 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:09:39.384126 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:39.384126 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:39.384139 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:39.384183 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:39.384466 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:39.384999 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:39.385905 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:39.386104 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:39.386158 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:39.388708 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55111: BlockingRecv error: recv got EOF from 127.0.98.2:55111 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 26 13:09:39 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:39 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:39.530682 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:39.525444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:36725)
0926 13:09:39.525619 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:39.525635 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:39.525712 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:39.525850 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:39.525851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:39.526020 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:39.526114 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:39.526116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:39.526799 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:39.526800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:39.527459 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:39.527462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:39.527887 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:39.527888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:39.527934 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:39.528465 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:39.528470 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:39.530558 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
W20260926 13:09:39.530725 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:39.525532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36725 (local address 127.0.98.62:35531)
0926 13:09:39.525695 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:39.525697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:39.525708 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:39.525772 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:39.525772 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:39.525783 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:39.525829 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:39.526178 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:39.526705 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:39.527558 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:39.527778 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:39.527876 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:39.530667 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36725: BlockingRecv error: recv got EOF from 127.0.98.3:36725 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Sep 26 13:09:40 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:40 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:40.259632 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:40.255097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:39809)
0926 13:09:40.255234 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:40.255246 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:40.255323 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:40.255568 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:40.255569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:40.255771 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:40.255817 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:40.255819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:40.256483 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:40.256484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:40.257021 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:40.257023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:40.257357 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:40.257358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:40.257408 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:40.257860 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:40.257866 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:40.259548 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
W20260926 13:09:40.259735 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:40.255285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39809 (local address 127.0.98.62:35531)
0926 13:09:40.255443 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:40.255444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:40.255453 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:40.255476 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:40.255477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:40.255489 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:40.255542 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:40.255893 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:40.256407 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:40.257104 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:40.257298 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:40.257352 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:40.259667 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39809: BlockingRecv error: recv got EOF from 127.0.98.1:39809 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
Sep 26 13:09:40 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:40 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:40.394289 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:40.389316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:45123)
0926 13:09:40.389491 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:40.389506 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:40.389590 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:40.389850 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:40.389851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:40.390058 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:40.390106 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:40.390107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:40.391135 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:40.391136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:40.391679 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:40.391681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:40.392088 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:40.392088 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:40.392130 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:40.392576 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:40.392581 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:40.394229 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
W20260926 13:09:40.394435 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:40.389427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45123 (local address 127.0.98.62:35531)
0926 13:09:40.389488 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:40.389491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:40.389587 (+ 96us) server_negotiation.cc:407] Connection header received
0926 13:09:40.389672 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:40.389672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:40.389688 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:40.389744 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:40.390186 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:40.391055 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:40.391824 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:40.392024 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:40.392082 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:40.394346 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45123: BlockingRecv error: recv got EOF from 127.0.98.2:45123 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 26 13:09:40 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:40 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:40.536666 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:40.531255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:33847)
0926 13:09:40.531435 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:40.531452 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:40.531550 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:40.531710 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:40.531711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:40.531912 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:40.531981 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:40.531983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:40.532827 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:40.532828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:40.533500 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:40.533503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:40.534018 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:40.534019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:40.534104 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:40.534625 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:40.534630 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:40.536589 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":62,"threads_started":1}
W20260926 13:09:40.536777 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:40.531404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33847 (local address 127.0.98.62:35531)
0926 13:09:40.531483 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:40.531484 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:40.531532 (+ 48us) server_negotiation.cc:407] Connection header received
0926 13:09:40.531621 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:40.531622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:40.531636 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:40.531691 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:40.532100 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:40.532688 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:40.533638 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:40.533909 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:40.533991 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:40.536696 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33847: BlockingRecv error: recv got EOF from 127.0.98.3:33847 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 13:09:41 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:41 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:41.265169 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:41.260327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:36409)
0926 13:09:41.260510 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:41.260522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:41.260582 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:41.260715 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:41.260716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:41.260883 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:41.260931 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:41.260932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:41.261653 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:41.261654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:41.262209 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:41.262212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:41.262562 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:41.262562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:41.262605 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:41.263091 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:41.263096 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:41.265098 (+ 2002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":93,"threads_started":1}
W20260926 13:09:41.265326 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:41.260458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36409 (local address 127.0.98.62:35531)
0926 13:09:41.260583 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:41.260585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:41.260616 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:09:41.260641 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:41.260642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:41.260653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:41.260713 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:41.261036 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:41.261547 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:41.262301 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:41.262494 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:41.262549 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:41.265239 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36409: BlockingRecv error: recv got EOF from 127.0.98.1:36409 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
Sep 26 13:09:41 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:41 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:41.399441 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:41.394754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:38559)
0926 13:09:41.394934 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:41.394948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:41.395018 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:41.395173 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:41.395175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:41.395347 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:41.395397 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:41.395399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:41.396092 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:41.396093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:41.396627 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:41.396629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:41.397077 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:41.397077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:41.397145 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:41.397615 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:41.397620 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:41.399381 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":72,"threads_started":1}
W20260926 13:09:41.399578 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:41.394907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38559 (local address 127.0.98.62:35531)
0926 13:09:41.394965 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:41.394967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:41.395002 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:09:41.395075 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:41.395076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:41.395087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:41.395125 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:41.395475 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:41.396006 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:41.396709 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:41.397005 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:41.397067 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:41.399506 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38559: BlockingRecv error: recv got EOF from 127.0.98.2:38559 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 13:09:41 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:41 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:41.541955 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:41.537250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:47241)
0926 13:09:41.537411 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:41.537423 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:41.537481 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:41.537629 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:41.537630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:41.537924 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:41.537994 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:41.537998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:41.538635 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:41.538636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:41.539267 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:41.539270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:41.539647 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:41.539648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:41.539690 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:41.540166 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:41.540171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:41.541894 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
W20260926 13:09:41.542048 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:41.537343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47241 (local address 127.0.98.62:35531)
0926 13:09:41.537393 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:41.537395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:41.537489 (+ 94us) server_negotiation.cc:407] Connection header received
0926 13:09:41.537540 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:41.537541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:41.537554 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:41.537607 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:41.538072 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:41.538560 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:41.539370 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:41.539593 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:41.539645 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:41.541984 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47241: BlockingRecv error: recv got EOF from 127.0.98.3:47241 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 13:09:42 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:42 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:42.271261 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:42.265831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:40993)
0926 13:09:42.266052 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:42.266070 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:42.266163 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:42.266452 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:42.266453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:42.266671 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:42.266741 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:42.266743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:42.267559 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:42.267560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:42.268194 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:42.268197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:42.268584 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:42.268584 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:42.268636 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:42.269227 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:42.269232 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:42.271186 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
W20260926 13:09:42.271405 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:42.266071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40993 (local address 127.0.98.62:35531)
0926 13:09:42.266281 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:42.266283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:42.266295 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:09:42.266321 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:42.266321 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:42.266332 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:42.266413 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:42.266879 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:42.267459 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:42.268289 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:42.268513 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:42.268577 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:42.271318 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40993: BlockingRecv error: recv got EOF from 127.0.98.1:40993 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
Sep 26 13:09:42 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:42 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:42.404838 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:42.400064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:60825)
0926 13:09:42.400225 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:42.400236 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:42.400301 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:42.400501 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:42.400502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:42.400655 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:42.400705 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:42.400707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:42.401492 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:42.401494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:42.402111 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:42.402113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:42.402492 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:42.402493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:42.402540 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:42.403154 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:42.403159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:42.404778 (+ 1619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
W20260926 13:09:42.405085 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:42.400123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60825 (local address 127.0.98.62:35531)
0926 13:09:42.400193 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:42.400195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:42.400303 (+ 108us) server_negotiation.cc:407] Connection header received
0926 13:09:42.400362 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:42.400363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:42.400375 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:42.400448 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:42.400794 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:42.401389 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:42.402206 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:42.402420 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:42.402476 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:42.405004 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60825: BlockingRecv error: recv got EOF from 127.0.98.2:60825 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 13:09:42 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:42 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:42.547598 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:42.542495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:37331)
0926 13:09:42.542658 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:42.542671 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:42.542743 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:42.542918 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:42.542919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:42.543129 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:42.543191 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:42.543193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:42.543892 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:42.543893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:42.544447 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:42.544449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:42.544782 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:42.544783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:42.544852 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:42.545358 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:42.545363 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:42.547504 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
W20260926 13:09:42.547679 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:42.542643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37331 (local address 127.0.98.62:35531)
0926 13:09:42.542732 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:42.542734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:42.542745 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:42.542802 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:42.542803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:42.542821 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:42.542898 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:42.543266 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:42.543820 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:42.544532 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:42.544722 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:42.544778 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:42.547616 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37331: BlockingRecv error: recv got EOF from 127.0.98.3:37331 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 13:09:43 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:43 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:43.276221 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:43.271769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:60875)
0926 13:09:43.271888 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:43.271899 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:43.271948 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:43.272100 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:43.272101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:43.272305 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:43.272361 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:43.272363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:43.273033 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:43.273034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:43.273567 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:43.273568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:43.273920 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:43.273921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:43.273957 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:43.274477 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:43.274482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:43.276157 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
W20260926 13:09:43.276371 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:43.271840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60875 (local address 127.0.98.62:35531)
0926 13:09:43.271958 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:43.271960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:43.271969 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:43.271994 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:43.271995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:43.272007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:43.272078 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:43.272429 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:43.272946 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:43.273670 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:43.273863 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:43.273912 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:43.276286 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60875: BlockingRecv error: recv got EOF from 127.0.98.1:60875 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
Sep 26 13:09:43 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:43 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:43.409706 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:43.405352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:45183)
0926 13:09:43.405464 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:43.405481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:43.405529 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:43.405674 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:43.405675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:43.405830 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:43.405875 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:43.405877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:43.406580 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:43.406581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:43.407179 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:43.407182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:43.407516 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:43.407516 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:43.407552 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:43.408010 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:43.408014 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:43.409645 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
W20260926 13:09:43.409821 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:43.405428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45183 (local address 127.0.98.62:35531)
0926 13:09:43.405507 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:43.405508 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:43.405521 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:09:43.405605 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:43.405606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:43.405617 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:43.405656 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:43.405930 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:43.406501 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:43.407264 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:43.407458 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:43.407515 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:43.409751 (+ 2236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45183: BlockingRecv error: recv got EOF from 127.0.98.2:45183 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 13:09:43 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:43 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:43.553005 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:43.548152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:36179)
0926 13:09:43.548330 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:43.548344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:43.548421 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:43.548564 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:43.548565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:43.548747 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:43.548800 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:43.548803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:43.549424 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:43.549425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:43.550006 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:43.550009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:43.550427 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:43.550428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:43.550486 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:43.551115 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:43.551120 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:43.552939 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
W20260926 13:09:43.553088 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:43.548274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36179 (local address 127.0.98.62:35531)
0926 13:09:43.548350 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:43.548352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:43.548410 (+ 58us) server_negotiation.cc:407] Connection header received
0926 13:09:43.548482 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:43.548483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:43.548496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:43.548553 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:43.548869 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:43.549353 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:43.550178 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:43.550368 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:43.550421 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:43.553031 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36179: BlockingRecv error: recv got EOF from 127.0.98.3:36179 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 13:09:44 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:44 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:44.291133 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:44.276859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:44865)
0926 13:09:44.277027 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:44.277042 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:44.277114 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:44.277312 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:44.277313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:44.277545 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:44.277615 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:44.277617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:44.278321 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:44.278322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:44.278963 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:44.278965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:44.279321 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:44.279322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:44.279364 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:44.288769 (+ 9405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:44.288777 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:44.291046 (+ 2269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
W20260926 13:09:44.291441 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:44.276993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44865 (local address 127.0.98.62:35531)
0926 13:09:44.277150 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:44.277152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:44.277164 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:09:44.277191 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:44.277192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:44.277202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:44.277294 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:44.277714 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:44.278234 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:44.279046 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:44.279249 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:44.279318 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:44.291379 (+ 12061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44865: BlockingRecv error: recv got EOF from 127.0.98.1:44865 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
Sep 26 13:09:44 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:44 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:44.416121 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:44.410239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:37833)
0926 13:09:44.410437 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:44.410453 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:44.410598 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:44.410814 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:44.410816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:44.411208 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:44.411294 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:44.411297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:44.412068 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:44.412069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:44.412770 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:44.412772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:44.413240 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:44.413243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:44.413363 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:44.413866 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:44.413871 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:44.415981 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
W20260926 13:09:44.416131 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:44.410402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37833 (local address 127.0.98.62:35531)
0926 13:09:44.410466 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:44.410468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:44.410587 (+ 119us) server_negotiation.cc:407] Connection header received
0926 13:09:44.410658 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:44.410659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:44.410669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:44.410745 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:44.411417 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:44.411956 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:44.412875 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:44.413105 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:44.413186 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:44.416079 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37833: BlockingRecv error: recv got EOF from 127.0.98.2:37833 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 13:09:44 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:44 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:44.558553 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:44.553524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:56763)
0926 13:09:44.553703 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:44.553717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:44.553788 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:44.553977 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:44.553977 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:44.554169 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:44.554235 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:44.554236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:44.554964 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:44.554965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:44.555555 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:44.555558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:44.555972 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:44.555973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:44.556013 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:44.556553 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:44.556558 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:44.558441 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
W20260926 13:09:44.558673 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:44.553620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56763 (local address 127.0.98.62:35531)
0926 13:09:44.553708 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:44.553710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:44.553791 (+ 81us) server_negotiation.cc:407] Connection header received
0926 13:09:44.553885 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:44.553885 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:44.553901 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:44.553958 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:44.554340 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:44.554878 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:44.555687 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:44.555890 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:44.555962 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:44.558600 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56763: BlockingRecv error: recv got EOF from 127.0.98.3:56763 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 13:09:45 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:45 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:45.296650 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:45.291734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:48339)
0926 13:09:45.291883 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:45.291931 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:45.291997 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:45.292195 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:45.292196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:45.292418 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:45.292486 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:45.292489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:45.293148 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:45.293149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:45.293688 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:45.293690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:45.294057 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:45.294058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:45.294098 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:45.294585 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:45.294589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:45.296580 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
W20260926 13:09:45.296772 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:45.291841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48339 (local address 127.0.98.62:35531)
0926 13:09:45.291973 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:45.291974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:45.291987 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:09:45.292056 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:45.292057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:45.292070 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:45.292134 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:45.292565 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:45.293071 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:45.293791 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:45.293990 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:45.294054 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:45.296695 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48339: BlockingRecv error: recv got EOF from 127.0.98.1:48339 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
Sep 26 13:09:45 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:45 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:45.421269 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:45.416679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:57761)
0926 13:09:45.416842 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:45.416854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:45.416930 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:45.417050 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:45.417051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:45.417205 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:45.417263 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:45.417265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:45.417922 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:45.417923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:45.418541 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:45.418543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:45.418992 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:45.418993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:45.419029 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:45.419504 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:45.419508 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:45.421200 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
W20260926 13:09:45.421365 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:45.416801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57761 (local address 127.0.98.62:35531)
0926 13:09:45.416869 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:45.416871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:45.416913 (+ 42us) server_negotiation.cc:407] Connection header received
0926 13:09:45.416979 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:45.416980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:45.416990 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:45.417036 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:45.417318 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:45.417837 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:45.418639 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:45.418833 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:45.418995 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:45.421302 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57761: BlockingRecv error: recv got EOF from 127.0.98.2:57761 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 13:09:45 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:45 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:45.563491 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:45.559147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:49025)
0926 13:09:45.559284 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:45.559294 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:45.559355 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:45.559491 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:45.559492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:45.559661 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:45.559713 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:45.559715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:45.560332 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:45.560334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:45.560872 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:45.560874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:45.561228 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:45.561229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:45.561283 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:45.561743 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:45.561747 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:45.563412 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
W20260926 13:09:45.563588 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:45.559303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49025 (local address 127.0.98.62:35531)
0926 13:09:45.559367 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:45.559369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:45.559378 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:09:45.559405 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:45.559406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:45.559421 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:45.559483 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:45.559784 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:45.560263 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:45.560957 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:45.561161 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:45.561219 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:45.563516 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49025: BlockingRecv error: recv got EOF from 127.0.98.3:49025 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 13:09:46 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:46 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:46.302912 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:46.297313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:56205)
0926 13:09:46.297500 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:46.297514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:46.297592 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:46.297790 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:46.297791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:46.297962 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:46.298012 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:46.298014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:46.298817 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:46.298818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:46.299429 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:46.299431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:46.299891 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:46.299892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:46.299950 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:46.300517 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:46.300521 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:46.302780 (+ 2259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
W20260926 13:09:46.303020 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:46.297452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56205 (local address 127.0.98.62:35531)
0926 13:09:46.297587 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:46.297589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:46.297600 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:46.297662 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:46.297663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:46.297675 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:46.297773 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:46.298105 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:46.298675 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:46.299530 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:46.299781 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:46.299844 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:46.302946 (+ 3102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56205: BlockingRecv error: recv got EOF from 127.0.98.1:56205 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":54,"threads_started":1}
Sep 26 13:09:46 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:46 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:46.427212 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:46.421899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:49141)
0926 13:09:46.422093 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:46.422109 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:46.422210 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:46.422441 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:46.422442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:46.422606 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:46.422652 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:46.422653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:46.423506 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:46.423507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:46.424039 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:46.424041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:46.424380 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:46.424381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:46.424431 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:46.424967 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:46.424972 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:46.427141 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
W20260926 13:09:46.427419 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:46.421987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49141 (local address 127.0.98.62:35531)
0926 13:09:46.422096 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:46.422099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:46.422186 (+ 87us) server_negotiation.cc:407] Connection header received
0926 13:09:46.422270 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:46.422271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:46.422284 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:46.422336 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:46.422740 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:46.423429 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:46.424118 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:46.424318 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:46.424373 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:46.427345 (+ 2972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49141: BlockingRecv error: recv got EOF from 127.0.98.2:49141 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 13:09:46 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:46 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:46.569589 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:46.564120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.3:54305)
0926 13:09:46.564336 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:46.564351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:46.564429 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:46.564597 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:46.564597 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:46.564797 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:46.564866 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:46.564868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:46.565605 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:46.565605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:46.566196 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:46.566198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:46.566678 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:46.566679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:46.566726 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:46.567297 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:46.567301 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:46.569503 (+ 2202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":116,"threads_started":1}
W20260926 13:09:46.569708 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:46.564238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54305 (local address 127.0.98.62:35531)
0926 13:09:46.564301 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:46.564303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:46.564425 (+ 122us) server_negotiation.cc:407] Connection header received
0926 13:09:46.564482 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:46.564482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:46.564494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:46.564573 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:46.564966 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:46.565524 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:46.566291 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:46.566573 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:46.566670 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:46.569629 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54305: BlockingRecv error: recv got EOF from 127.0.98.3:54305 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 13:09:47 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:47 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:47.308503 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:47.303673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.1:48631)
0926 13:09:47.303826 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:47.303839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:47.303913 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:47.304093 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:47.304095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:47.304255 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:47.304306 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:47.304308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.305134 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.305135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:47.305697 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:47.305700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.306093 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.306094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:47.306153 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:47.306655 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:47.306660 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:47.308429 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
W20260926 13:09:47.308691 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:47.303798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48631 (local address 127.0.98.62:35531)
0926 13:09:47.303914 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:47.303916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:47.303946 (+ 30us) server_negotiation.cc:407] Connection header received
0926 13:09:47.303971 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:47.303972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:47.303982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:47.304031 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:47.304407 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.305044 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.305792 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.306015 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.306081 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:47.308615 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48631: BlockingRecv error: recv got EOF from 127.0.98.1:48631 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":51,"threads_started":1}
Sep 26 13:09:47 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:09:47 dist-test-slave-2dw5 krb5kdc[8531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:09:47.432770 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:47.427875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35531 (local address 127.0.98.2:54647)
0926 13:09:47.428032 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:47.428042 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:47.428099 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:47.428292 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:47.428293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:47.428454 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:47.428502 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:47.428504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.429164 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.429166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:47.429683 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:47.429685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.430012 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.430013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:47.430063 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:47.430564 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:47.430569 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:47.432701 (+ 2132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35531: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
W20260926 13:09:47.432853 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:47.428037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54647 (local address 127.0.98.62:35531)
0926 13:09:47.428106 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:47.428108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:47.428118 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:47.428220 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:47.428221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:47.428233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:47.428276 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:47.428574 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.429082 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.429767 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.429953 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.430008 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:47.432796 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54647: BlockingRecv error: recv got EOF from 127.0.98.2:54647 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:09:47.475385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 9115
I20260926 13:09:47.479687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 9266
I20260926 13:09:47.483651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 9398
I20260926 13:09:47.487717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 9044
2026-09-26T13:09:47Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17389 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T13:09:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:47Z Disabled control of system clock
I20260926 13:09:47.513689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40711
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 13:09:47.593307 9594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:47.593541 9594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:47.593570 9594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:47.594825 9594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:47.594901 9594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:47.594918 9594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:47.594933 9594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:47.594949 9594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:47.596429 9594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:47.596694 9594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:47.596976 9594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:47.602654 9599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:47.602730 9594 server_base.cc:1034] running on GCE node
W20260926 13:09:47.602612 9602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:47.602684 9600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:47.603116 9594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:47.603416 9594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:47.604586 9594 hybrid_clock.cc:648] HybridClock initialized: now 1790428187604610 us; error 67 us; skew 500 ppm
I20260926 13:09:47.605983 9594 webserver.cc:460] Webserver started at http://127.0.98.62:35013/ using document root <none> and password file <none>
I20260926 13:09:47.606235 9594 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:47.606292 9594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:47.606446 9594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:47.607770 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "fb0850782a5240dd989bf6034ac3d237"
format_stamp: "Formatted at 2026-09-26 13:09:47 on dist-test-slave-2dw5"
server_key: "c90dc417477fa5f382539203ab8258ec"
server_key_iv: "b405a3ab3dcd89f370e3052e19d6ab9b"
server_key_version: "encryptionkey@0"
I20260926 13:09:47.608194 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb0850782a5240dd989bf6034ac3d237"
format_stamp: "Formatted at 2026-09-26 13:09:47 on dist-test-slave-2dw5"
server_key: "c90dc417477fa5f382539203ab8258ec"
server_key_iv: "b405a3ab3dcd89f370e3052e19d6ab9b"
server_key_version: "encryptionkey@0"
I20260926 13:09:47.609596 9594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:09:47.610575 9608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:47.610963 9594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:47.611085 9594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "fb0850782a5240dd989bf6034ac3d237"
format_stamp: "Formatted at 2026-09-26 13:09:47 on dist-test-slave-2dw5"
server_key: "c90dc417477fa5f382539203ab8258ec"
server_key_iv: "b405a3ab3dcd89f370e3052e19d6ab9b"
server_key_version: "encryptionkey@0"
I20260926 13:09:47.611174 9594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:47.624925 9594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:47.625250 9594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:47.625376 9594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:47.629115 9594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40711
I20260926 13:09:47.629155 9660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40711 every 8 connection(s)
I20260926 13:09:47.629470 9594 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:47.630226 9661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:47.632920 9661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237: Bootstrap starting.
I20260926 13:09:47.633518 9661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:47.633783 9661 log.cc:824] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:47.634495 9661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237: No bootstrap required, opened a new log
I20260926 13:09:47.636116 9661 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0850782a5240dd989bf6034ac3d237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40711 } }
I20260926 13:09:47.636309 9661 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:47.636332 9661 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0850782a5240dd989bf6034ac3d237, State: Initialized, Role: FOLLOWER
I20260926 13:09:47.636473 9661 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0850782a5240dd989bf6034ac3d237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40711 } }
I20260926 13:09:47.636549 9661 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:47.636574 9661 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:47.636618 9661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:47.637126 9661 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0850782a5240dd989bf6034ac3d237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40711 } }
I20260926 13:09:47.637229 9661 leader_election.cc:302] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0850782a5240dd989bf6034ac3d237; no voters:
I20260926 13:09:47.637408 9661 leader_election.cc:288] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:47.637511 9664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:47.637629 9661 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:47.637703 9664 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [term 1 LEADER]: Becoming Leader. State: Replica: fb0850782a5240dd989bf6034ac3d237, State: Running, Role: LEADER
I20260926 13:09:47.637799 9664 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0850782a5240dd989bf6034ac3d237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40711 } }
I20260926 13:09:47.638111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 9594
I20260926 13:09:47.638202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:47.638370 392 external_mini_cluster.cc:1428] Setting key e327ee3d6d558fd9a879b82981a872c6
I20260926 13:09:47.639359 9666 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0850782a5240dd989bf6034ac3d237. Latest consensus state: current_term: 1 leader_uuid: "fb0850782a5240dd989bf6034ac3d237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0850782a5240dd989bf6034ac3d237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40711 } } }
I20260926 13:09:47.639447 9665 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb0850782a5240dd989bf6034ac3d237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0850782a5240dd989bf6034ac3d237" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40711 } } }
I20260926 13:09:47.639569 9666 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:47.639599 9665 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237 [sys.catalog]: This master's current role is: LEADER
W20260926 13:09:47.640055 9680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:09:47.640110 9680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:09:47.640216 9681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:47.640378 9681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:47.641903 9681 catalog_manager.cc:1347] Generated new cluster ID: 7ea8bb7014584477b3f60204fd74ae92
I20260926 13:09:47.641956 9681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:47.644212 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:47.638835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.0.98.62:40711)
0926 13:09:47.639081 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:47.639089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:47.639125 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:09:47.639249 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:47.639257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:47.639327 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:47.639763 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:47.639860 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.641133 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.642124 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.642730 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.642886 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:47.643518 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:47.643522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:47.643522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:47.643726 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:47.643770 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:47.643785 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:47.643856 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:47.643912 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 13:09:47.643992 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260926 13:09:47.646510 9681 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:47.647238 9681 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:47.662492 9681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb0850782a5240dd989bf6034ac3d237: Generated new TSK 0
I20260926 13:09:47.662714 9681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:09:47.665006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40711
--builtin_ntp_servers=127.0.98.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 13:09:47.748901 9685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:47.749142 9685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:47.749166 9685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:47.750479 9685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:47.750545 9685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:47.750573 9685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:47.752115 9685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:47.752390 9685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:47.752712 9685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:47.757891 9691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:47.757887 9690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:47.758239 9685 server_base.cc:1034] running on GCE node
W20260926 13:09:47.758258 9693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:47.758498 9685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:47.758769 9685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:47.759932 9685 hybrid_clock.cc:648] HybridClock initialized: now 1790428187759913 us; error 33 us; skew 500 ppm
I20260926 13:09:47.761423 9685 webserver.cc:460] Webserver started at http://127.0.98.1:39133/ using document root <none> and password file <none>
I20260926 13:09:47.761708 9685 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:47.761770 9685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:47.761898 9685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:47.763365 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "e244a16c59824354b7612b56a3388d9b"
format_stamp: "Formatted at 2026-09-26 13:09:47 on dist-test-slave-2dw5"
server_key: "69da1326cc5c60dfcaa2aa3b5d4aa38e"
server_key_iv: "ad4cff49f9b8f20df223867b6426b8f5"
server_key_version: "encryptionkey@0"
I20260926 13:09:47.763765 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e244a16c59824354b7612b56a3388d9b"
format_stamp: "Formatted at 2026-09-26 13:09:47 on dist-test-slave-2dw5"
server_key: "69da1326cc5c60dfcaa2aa3b5d4aa38e"
server_key_iv: "ad4cff49f9b8f20df223867b6426b8f5"
server_key_version: "encryptionkey@0"
I20260926 13:09:47.765430 9685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:47.766390 9699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:47.766582 9685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:47.766680 9685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "e244a16c59824354b7612b56a3388d9b"
format_stamp: "Formatted at 2026-09-26 13:09:47 on dist-test-slave-2dw5"
server_key: "69da1326cc5c60dfcaa2aa3b5d4aa38e"
server_key_iv: "ad4cff49f9b8f20df223867b6426b8f5"
server_key_version: "encryptionkey@0"
I20260926 13:09:47.766763 9685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:47.795384 9685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:47.795675 9685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:47.795780 9685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:47.795970 9685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:47.796314 9685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:47.796345 9685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:47.796367 9685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:47.796388 9685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:47.801801 9685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38149
I20260926 13:09:47.801852 9812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38149 every 8 connection(s)
I20260926 13:09:47.802181 9685 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:09:47.807267 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:47.802884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38655 (local address 127.0.98.62:40711)
0926 13:09:47.802926 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:47.802928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:47.803180 (+ 252us) server_negotiation.cc:407] Connection header received
0926 13:09:47.803718 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:47.803720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:47.803734 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:47.803782 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:47.804439 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.804946 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.805905 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.806137 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.806190 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:47.806999 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:47.807001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:47.807001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:47.807006 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:47.807021 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:47.807025 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:47.807065 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:47.807141 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 13:09:47.807173 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:09:47.807463 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:47.802881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40711 (local address 127.0.98.1:38655)
0926 13:09:47.803070 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:47.803103 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:47.803622 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:47.803863 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:47.803867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:47.803875 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:47.804358 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:47.804363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.805041 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.805042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:47.805824 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:47.805826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.806212 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.806213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:47.806334 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:47.806859 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:47.806860 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:47.806895 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:47.806918 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:47.807070 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:47.807075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:47.807078 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:47.807140 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:09:47.807252 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260926 13:09:47.807878 9813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40711
I20260926 13:09:47.807960 9813 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:47.808139 9813 heartbeater.cc:511] Master 127.0.98.62:40711 requested a full tablet report, sending...
I20260926 13:09:47.808610 9625 ts_manager.cc:194] Registered new tserver with Master: e244a16c59824354b7612b56a3388d9b (127.0.98.1:38149)
I20260926 13:09:47.809190 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38655
I20260926 13:09:47.810178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 9685
I20260926 13:09:47.810259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:47.810401 392 external_mini_cluster.cc:1428] Setting key 43f0390ce6764af5e0888011776089a4
I20260926 13:09:47.811344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40711
--builtin_ntp_servers=127.0.98.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 13:09:47.894986 9816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:47.895207 9816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:47.895237 9816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:47.896454 9816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:47.896505 9816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:47.896539 9816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:47.897905 9816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:47.898160 9816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:47.898442 9816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:47.903479 9822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:47.903499 9824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:47.903482 9821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:47.903738 9816 server_base.cc:1034] running on GCE node
I20260926 13:09:47.903957 9816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:47.904187 9816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:47.905357 9816 hybrid_clock.cc:648] HybridClock initialized: now 1790428187905343 us; error 30 us; skew 500 ppm
I20260926 13:09:47.906638 9816 webserver.cc:460] Webserver started at http://127.0.98.2:39565/ using document root <none> and password file <none>
I20260926 13:09:47.906883 9816 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:47.906937 9816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:47.907063 9816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:47.908229 9816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "94f029ee9907483a8b856c4fd0fe6c5d"
format_stamp: "Formatted at 2026-09-26 13:09:47 on dist-test-slave-2dw5"
server_key: "741a765c170e2fb012284aba9c1a32d2"
server_key_iv: "27c64492ba93620739c6e96d83f50422"
server_key_version: "encryptionkey@0"
I20260926 13:09:47.908583 9816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "94f029ee9907483a8b856c4fd0fe6c5d"
format_stamp: "Formatted at 2026-09-26 13:09:47 on dist-test-slave-2dw5"
server_key: "741a765c170e2fb012284aba9c1a32d2"
server_key_iv: "27c64492ba93620739c6e96d83f50422"
server_key_version: "encryptionkey@0"
I20260926 13:09:47.909945 9816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:09:47.910876 9830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:47.911037 9816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:09:47.911144 9816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "94f029ee9907483a8b856c4fd0fe6c5d"
format_stamp: "Formatted at 2026-09-26 13:09:47 on dist-test-slave-2dw5"
server_key: "741a765c170e2fb012284aba9c1a32d2"
server_key_iv: "27c64492ba93620739c6e96d83f50422"
server_key_version: "encryptionkey@0"
I20260926 13:09:47.911226 9816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:47.927647 9816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:47.927976 9816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:47.928119 9816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:47.928344 9816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:47.928664 9816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:47.928697 9816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:47.928726 9816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:47.928750 9816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:47.934240 9816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40025
I20260926 13:09:47.934293 9943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40025 every 8 connection(s)
I20260926 13:09:47.934617 9816 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:09:47.935875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 9816
I20260926 13:09:47.935958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:47.936151 392 external_mini_cluster.cc:1428] Setting key 5e305c763d24059a38026090b63018f8
I20260926 13:09:47.937137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40711
--builtin_ntp_servers=127.0.98.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 13:09:47.940075 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:47.935347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45927 (local address 127.0.98.62:40711)
0926 13:09:47.935422 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:47.935424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:47.935591 (+ 167us) server_negotiation.cc:407] Connection header received
0926 13:09:47.936105 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:47.936107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:47.936121 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:47.936195 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:47.936907 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.937480 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.938625 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.938816 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.938922 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:47.939819 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:47.939820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:47.939821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:47.939826 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:47.939842 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:47.939847 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:47.939890 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:47.939962 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 13:09:47.939992 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:09:47.940505 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:47.935290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40711 (local address 127.0.98.2:45927)
0926 13:09:47.935472 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:47.935513 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:47.935998 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:47.936288 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:47.936292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:47.936303 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:47.936806 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:47.936810 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.937567 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.937568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:47.938554 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:47.938557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:47.938948 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:47.938949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:47.939121 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:47.939679 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:47.939679 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:47.939714 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:47.939740 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:47.939891 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:47.939896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:47.939900 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:47.939963 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:09:47.940344 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260926 13:09:47.940941 9944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40711
I20260926 13:09:47.941030 9944 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:47.941211 9944 heartbeater.cc:511] Master 127.0.98.62:40711 requested a full tablet report, sending...
I20260926 13:09:47.941568 9625 ts_manager.cc:194] Registered new tserver with Master: 94f029ee9907483a8b856c4fd0fe6c5d (127.0.98.2:40025)
I20260926 13:09:47.942080 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45927
W20260926 13:09:48.019726 9947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:48.019936 9947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:48.019958 9947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:48.021147 9947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:48.021190 9947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:48.021214 9947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:48.022611 9947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:48.022832 9947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:48.023217 9947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:09:48.028617 9947 server_base.cc:1034] running on GCE node
W20260926 13:09:48.028604 9952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:48.028606 9953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:48.028777 9955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:48.028955 9947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:48.029211 9947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:48.030362 9947 hybrid_clock.cc:648] HybridClock initialized: now 1790428188030345 us; error 36 us; skew 500 ppm
I20260926 13:09:48.031791 9947 webserver.cc:460] Webserver started at http://127.0.98.3:46189/ using document root <none> and password file <none>
I20260926 13:09:48.032015 9947 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:48.032066 9947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:48.032192 9947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:48.033262 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "a129ad702de643269abd35591a27b42d"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "057b887bcb5642b6b59218791ccaac92"
server_key_iv: "8d536c1bb197f24f28ff622254d5e134"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.033600 9947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a129ad702de643269abd35591a27b42d"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "057b887bcb5642b6b59218791ccaac92"
server_key_iv: "8d536c1bb197f24f28ff622254d5e134"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.034963 9947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:48.035799 9961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.036015 9947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:48.036108 9947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "a129ad702de643269abd35591a27b42d"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "057b887bcb5642b6b59218791ccaac92"
server_key_iv: "8d536c1bb197f24f28ff622254d5e134"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.036187 9947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:48.044914 9947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:48.045228 9947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:48.045359 9947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:48.045583 9947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:48.045900 9947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:48.045933 9947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.045962 9947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:48.045979 9947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.051685 9947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44487
I20260926 13:09:48.051761 10074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44487 every 8 connection(s)
I20260926 13:09:48.052073 9947 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:48.057287 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:48.052771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57827 (local address 127.0.98.62:40711)
0926 13:09:48.052843 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:48.052845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:48.053086 (+ 241us) server_negotiation.cc:407] Connection header received
0926 13:09:48.053625 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:48.053626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:48.053637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:48.053702 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:48.054408 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:48.054969 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:48.055922 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:48.056127 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:48.056181 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:48.056988 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:48.056989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:48.056989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:48.056994 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:48.057009 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:48.057013 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:48.057066 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:48.057196 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 13:09:48.057222 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:09:48.057502 10077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:48.052747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40711 (local address 127.0.98.3:57827)
0926 13:09:48.052927 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:48.052971 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:48.053505 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:48.053782 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:48.053786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:48.053795 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:48.054318 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:48.054322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:48.055063 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:48.055064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:48.055840 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:48.055842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:48.056204 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:48.056204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:48.056316 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:48.056824 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:48.056824 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:48.056855 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:48.056879 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:48.057104 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:48.057110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:48.057118 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 13:09:48.057192 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 13:09:48.057288 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260926 13:09:48.058024 10075 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40711
I20260926 13:09:48.058112 10075 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:48.058264 10075 heartbeater.cc:511] Master 127.0.98.62:40711 requested a full tablet report, sending...
I20260926 13:09:48.058588 9625 ts_manager.cc:194] Registered new tserver with Master: a129ad702de643269abd35591a27b42d (127.0.98.3:44487)
I20260926 13:09:48.058971 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57827
I20260926 13:09:48.062044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 9947
I20260926 13:09:48.062124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:48.062294 392 external_mini_cluster.cc:1428] Setting key 2f51a251e17c689c9fb8325336e086b8
I20260926 13:09:48.063297 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 13:09:48.065117 9678 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:48.064608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.0.98.62:40711)
0926 13:09:48.064668 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:48.064670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:48.064793 (+ 123us) server_negotiation.cc:407] Connection header received
0926 13:09:48.064848 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:48.064849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:48.064861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:48.064906 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:48.065039 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36052: BlockingRecv error: recv got EOF from 127.0.0.1:36052 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:09:48.065157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 9685
I20260926 13:09:48.069284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 9816
I20260926 13:09:48.073218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 9947
I20260926 13:09:48.077224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 9594
2026-09-26T13:09:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (589 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T13:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:48Z Disabled control of system clock
I20260926 13:09:48.102648 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36019
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:09:48.184078 10086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:48.184289 10086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:48.184309 10086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:48.185479 10086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:48.185530 10086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:48.185542 10086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:48.185554 10086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:48.185564 10086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:48.187011 10086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:48.187271 10086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:48.187557 10086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:48.192160 10094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:48.192157 10092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:48.192162 10091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:48.192457 10086 server_base.cc:1034] running on GCE node
I20260926 13:09:48.192729 10086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:48.193001 10086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:48.194160 10086 hybrid_clock.cc:648] HybridClock initialized: now 1790428188194148 us; error 29 us; skew 500 ppm
I20260926 13:09:48.203359 10086 webserver.cc:460] Webserver started at http://127.0.98.62:45621/ using document root <none> and password file <none>
I20260926 13:09:48.203620 10086 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:48.203691 10086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:48.203819 10086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:48.204898 10086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "b02d4129c17643cd92fba89ae10aa3d8"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "a3f702b320267fefe80a5470bc3cc0f8"
server_key_iv: "c84f7b2299f24f1ef4afe8994a8dc659"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.205224 10086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "b02d4129c17643cd92fba89ae10aa3d8"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "a3f702b320267fefe80a5470bc3cc0f8"
server_key_iv: "c84f7b2299f24f1ef4afe8994a8dc659"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.206411 10086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:48.207139 10100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.207294 10086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:09:48.207376 10086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "b02d4129c17643cd92fba89ae10aa3d8"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "a3f702b320267fefe80a5470bc3cc0f8"
server_key_iv: "c84f7b2299f24f1ef4afe8994a8dc659"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.207433 10086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:48.210825 10086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:48.211134 10086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:48.211242 10086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:48.215157 10086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36019
I20260926 13:09:48.215188 10152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36019 every 8 connection(s)
I20260926 13:09:48.215560 10086 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:48.216217 10153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:48.216972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 10086
I20260926 13:09:48.217073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:48.217254 392 external_mini_cluster.cc:1428] Setting key 89dd28990a0c55c5c2207e5a9616ead2
I20260926 13:09:48.219796 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:48.217803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.0.98.62:36019)
0926 13:09:48.218007 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:48.218016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:48.218064 (+ 48us) server_negotiation.cc:407] Connection header received
0926 13:09:48.218528 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:48.218535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:48.218581 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:48.218702 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:48.219375 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:48.219378 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:48.219378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:48.219381 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:48.219420 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:48.219431 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:48.219485 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:48.219553 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 13:09:48.219612 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260926 13:09:48.220330 10153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8: Bootstrap starting.
I20260926 13:09:48.220937 10153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:48.221199 10153 log.cc:824] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:48.221906 10153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8: No bootstrap required, opened a new log
I20260926 13:09:48.223322 10153 raft_consensus.cc:348] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d4129c17643cd92fba89ae10aa3d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36019 } }
I20260926 13:09:48.223498 10153 raft_consensus.cc:374] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:48.223541 10153 raft_consensus.cc:729] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02d4129c17643cd92fba89ae10aa3d8, State: Initialized, Role: FOLLOWER
I20260926 13:09:48.223675 10153 consensus_queue.cc:260] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d4129c17643cd92fba89ae10aa3d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36019 } }
I20260926 13:09:48.223764 10153 raft_consensus.cc:388] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:48.223794 10153 raft_consensus.cc:482] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:48.223845 10153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:48.224417 10153 raft_consensus.cc:504] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d4129c17643cd92fba89ae10aa3d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36019 } }
I20260926 13:09:48.224505 10153 leader_election.cc:302] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b02d4129c17643cd92fba89ae10aa3d8; no voters:
I20260926 13:09:48.224666 10153 leader_election.cc:288] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:48.224912 10158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:48.224922 10153 sys_catalog.cc:564] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:48.225087 10158 raft_consensus.cc:686] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [term 1 LEADER]: Becoming Leader. State: Replica: b02d4129c17643cd92fba89ae10aa3d8, State: Running, Role: LEADER
I20260926 13:09:48.225191 10158 consensus_queue.cc:237] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d4129c17643cd92fba89ae10aa3d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36019 } }
I20260926 13:09:48.225512 10159 sys_catalog.cc:455] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b02d4129c17643cd92fba89ae10aa3d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d4129c17643cd92fba89ae10aa3d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36019 } } }
I20260926 13:09:48.225528 10160 sys_catalog.cc:455] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b02d4129c17643cd92fba89ae10aa3d8. Latest consensus state: current_term: 1 leader_uuid: "b02d4129c17643cd92fba89ae10aa3d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d4129c17643cd92fba89ae10aa3d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36019 } } }
I20260926 13:09:48.225631 10159 sys_catalog.cc:458] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:48.225641 10160 sys_catalog.cc:458] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:48.225848 10167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:48.226332 10167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:48.227859 10167 catalog_manager.cc:1347] Generated new cluster ID: 0e00c90b3c72405c89fb6faaa061a2dc
I20260926 13:09:48.227916 10167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:48.247349 10167 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:48.247556 10167 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:48.257750 10167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b02d4129c17643cd92fba89ae10aa3d8: Generated new TSK 0
I20260926 13:09:48.257966 10167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:09:48.264047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36019
--builtin_ntp_servers=127.0.98.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:09:48.347895 10177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:48.348109 10177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:48.348131 10177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:48.349365 10177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:48.349409 10177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:48.349435 10177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:48.350801 10177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:48.351071 10177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:48.351349 10177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:48.356441 10185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:48.356494 10183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:48.356815 10177 server_base.cc:1034] running on GCE node
W20260926 13:09:48.356451 10182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:48.357067 10177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:48.357362 10177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:48.358541 10177 hybrid_clock.cc:648] HybridClock initialized: now 1790428188358520 us; error 37 us; skew 500 ppm
I20260926 13:09:48.359964 10177 webserver.cc:460] Webserver started at http://127.0.98.1:46763/ using document root <none> and password file <none>
I20260926 13:09:48.360214 10177 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:48.360272 10177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:48.360399 10177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:48.361622 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "e731e2c270054d308140bfa5a2a75dcd"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "c4021b634e6fc545c8fccb457a99a886"
server_key_iv: "989bb8e7d2a8892b28abfccb015c19de"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.361948 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e731e2c270054d308140bfa5a2a75dcd"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "c4021b634e6fc545c8fccb457a99a886"
server_key_iv: "989bb8e7d2a8892b28abfccb015c19de"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.363140 10177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:09:48.363895 10191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.364125 10177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:09:48.364215 10177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "e731e2c270054d308140bfa5a2a75dcd"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "c4021b634e6fc545c8fccb457a99a886"
server_key_iv: "989bb8e7d2a8892b28abfccb015c19de"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.364280 10177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:48.367823 10177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:48.368078 10177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:48.368184 10177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:48.368351 10177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:48.368607 10177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:48.368635 10177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.368659 10177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:48.368675 10177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.374477 10177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37571
I20260926 13:09:48.374536 10304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37571 every 8 connection(s)
I20260926 13:09:48.374874 10177 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:09:48.377916 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:48.375587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41803 (local address 127.0.98.62:36019)
0926 13:09:48.375666 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:48.375668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:48.375881 (+ 213us) server_negotiation.cc:407] Connection header received
0926 13:09:48.376360 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:48.376361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:48.376367 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:48.376420 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:48.377619 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:48.377620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:48.377621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:48.377624 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:48.377638 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:48.377643 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:48.377699 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:48.377847 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 13:09:48.377861 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:09:48.378044 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:48.375591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36019 (local address 127.0.98.1:41803)
0926 13:09:48.375770 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:48.375802 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:48.376256 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:48.376478 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:48.376481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:48.376489 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:48.377352 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:48.377353 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:48.377382 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:48.377410 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:48.377731 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:48.377737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:48.377740 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:48.377823 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 13:09:48.377876 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260926 13:09:48.378530 10305 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36019
I20260926 13:09:48.378616 10305 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:48.378739 10305 heartbeater.cc:511] Master 127.0.98.62:36019 requested a full tablet report, sending...
I20260926 13:09:48.379019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 10177
I20260926 13:09:48.379104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:48.379278 10117 ts_manager.cc:194] Registered new tserver with Master: e731e2c270054d308140bfa5a2a75dcd (127.0.98.1:37571)
I20260926 13:09:48.379298 392 external_mini_cluster.cc:1428] Setting key ee2831496445ef6fe2d6e16f50b382ac
I20260926 13:09:48.380076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36019
--builtin_ntp_servers=127.0.98.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:09:48.470963 10308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:48.471204 10308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:48.471230 10308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:48.472500 10308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:48.472550 10308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:48.472579 10308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:48.474059 10308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:48.474337 10308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:48.474635 10308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:48.479735 10314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:48.479743 10313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:48.479743 10316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:48.479890 10308 server_base.cc:1034] running on GCE node
I20260926 13:09:48.480228 10308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:48.480476 10308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:48.481657 10308 hybrid_clock.cc:648] HybridClock initialized: now 1790428188481640 us; error 29 us; skew 500 ppm
I20260926 13:09:48.483007 10308 webserver.cc:460] Webserver started at http://127.0.98.2:44371/ using document root <none> and password file <none>
I20260926 13:09:48.483254 10308 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:48.483304 10308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:48.483429 10308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:48.484495 10308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "4727102979ac4f6982c68886cdf7e3c0"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "0c25b7a077a21f0bb7ddaa0ab8398cf2"
server_key_iv: "cb974c9796af21b49a783bcac8982ce5"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.484831 10308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4727102979ac4f6982c68886cdf7e3c0"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "0c25b7a077a21f0bb7ddaa0ab8398cf2"
server_key_iv: "cb974c9796af21b49a783bcac8982ce5"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.486020 10308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:48.486692 10322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.486884 10308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:48.486970 10308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "4727102979ac4f6982c68886cdf7e3c0"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "0c25b7a077a21f0bb7ddaa0ab8398cf2"
server_key_iv: "cb974c9796af21b49a783bcac8982ce5"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.487033 10308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:48.490270 10308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:48.490522 10308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:48.490640 10308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:48.490818 10308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:48.491134 10308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:48.491168 10308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.491194 10308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:48.491210 10308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.496692 10308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44265
I20260926 13:09:48.496747 10435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44265 every 8 connection(s)
I20260926 13:09:48.497040 10308 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:09:48.500130 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:48.497756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33537 (local address 127.0.98.62:36019)
0926 13:09:48.497830 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:48.497832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:48.498020 (+ 188us) server_negotiation.cc:407] Connection header received
0926 13:09:48.498574 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:48.498646 (+ 72us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:48.498652 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:48.498713 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:48.499937 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:48.499938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:48.499938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:48.499943 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:48.499957 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:48.499961 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:48.500015 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:48.500066 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 13:09:48.500077 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:09:48.500309 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:48.497724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36019 (local address 127.0.98.2:33537)
0926 13:09:48.497912 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:48.497945 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:48.498399 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:48.498793 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:48.498797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:48.498805 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:48.499795 (+ 990us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:48.499796 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:48.499824 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:48.499852 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:48.500019 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:48.500021 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:48.500023 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:09:48.500073 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 13:09:48.500121 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260926 13:09:48.500823 10436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36019
I20260926 13:09:48.500913 10436 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:48.501060 10436 heartbeater.cc:511] Master 127.0.98.62:36019 requested a full tablet report, sending...
I20260926 13:09:48.501387 10117 ts_manager.cc:194] Registered new tserver with Master: 4727102979ac4f6982c68886cdf7e3c0 (127.0.98.2:44265)
I20260926 13:09:48.504842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 10308
I20260926 13:09:48.504940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:48.505151 392 external_mini_cluster.cc:1428] Setting key 260f9d8a5d8835219df780209213a6d8
I20260926 13:09:48.506068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36019
--builtin_ntp_servers=127.0.98.20:45643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:09:48.586409 10439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:48.586624 10439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:48.586648 10439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:48.587999 10439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:48.588057 10439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:48.588095 10439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:48.589540 10439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:48.589767 10439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:48.590044 10439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:48.595172 10445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:48.595160 10444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:48.595227 10439 server_base.cc:1034] running on GCE node
W20260926 13:09:48.595161 10447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:48.595607 10439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:48.595866 10439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:48.597038 10439 hybrid_clock.cc:648] HybridClock initialized: now 1790428188597021 us; error 28 us; skew 500 ppm
I20260926 13:09:48.598425 10439 webserver.cc:460] Webserver started at http://127.0.98.3:33745/ using document root <none> and password file <none>
I20260926 13:09:48.598680 10439 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:48.598737 10439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:48.598892 10439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:48.599956 10439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "7921e300c0c44218be71223e6ce6f3e5"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "8b7dc92252395b0b74db9d958bf5b728"
server_key_iv: "a5a2d8ffdc1620da4e7d635d6213fbfa"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.600296 10439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7921e300c0c44218be71223e6ce6f3e5"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "8b7dc92252395b0b74db9d958bf5b728"
server_key_iv: "a5a2d8ffdc1620da4e7d635d6213fbfa"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.601545 10439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:09:48.602303 10453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.602486 10439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:09:48.602586 10439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "7921e300c0c44218be71223e6ce6f3e5"
format_stamp: "Formatted at 2026-09-26 13:09:48 on dist-test-slave-2dw5"
server_key: "8b7dc92252395b0b74db9d958bf5b728"
server_key_iv: "a5a2d8ffdc1620da4e7d635d6213fbfa"
server_key_version: "encryptionkey@0"
I20260926 13:09:48.602658 10439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:48.606222 10439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:48.606485 10439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:48.606604 10439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:48.606798 10439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:48.607127 10439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:48.607160 10439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.607185 10439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:48.607211 10439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:48.612821 10439 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35063
I20260926 13:09:48.612869 10566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35063 every 8 connection(s)
I20260926 13:09:48.613186 10439 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:48.616295 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:48.613848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45437 (local address 127.0.98.62:36019)
0926 13:09:48.613890 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:48.613893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:48.614127 (+ 234us) server_negotiation.cc:407] Connection header received
0926 13:09:48.614690 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:48.614691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:48.614697 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:48.614763 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:48.616021 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:48.616023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:48.616023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:09:48.616027 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:48.616042 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:09:48.616046 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:48.616105 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:48.616228 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 13:09:48.616241 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:09:48.616473 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:48.613826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36019 (local address 127.0.98.3:45437)
0926 13:09:48.613998 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:48.614036 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:48.614473 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:48.614904 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:48.614909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:48.614920 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:48.615869 (+ 949us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:09:48.615870 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:48.615899 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:09:48.615929 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:48.616137 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:48.616142 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:48.616145 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:48.616225 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 13:09:48.616277 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260926 13:09:48.617002 10567 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36019
I20260926 13:09:48.617079 10567 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:48.617195 10567 heartbeater.cc:511] Master 127.0.98.62:36019 requested a full tablet report, sending...
I20260926 13:09:48.617507 10117 ts_manager.cc:194] Registered new tserver with Master: 7921e300c0c44218be71223e6ce6f3e5 (127.0.98.3:35063)
I20260926 13:09:48.620738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 10439
I20260926 13:09:48.620815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:48.620965 392 external_mini_cluster.cc:1428] Setting key a157e308781371215ef1b7bfa1df9d02
I20260926 13:09:48.622044 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 13:09:48.623731 10156 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:09:48.623338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.0.98.62:36019)
0926 13:09:48.623398 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:48.623400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:48.623455 (+ 55us) server_negotiation.cc:407] Connection header received
0926 13:09:48.623507 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:48.623508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:48.623512 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:48.623555 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:48.623668 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43366: BlockingRecv error: recv got EOF from 127.0.0.1:43366 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:09:48.623884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 10177
I20260926 13:09:48.628414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 10308
I20260926 13:09:48.632568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 10439
I20260926 13:09:48.636066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 10086
2026-09-26T13:09:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (558 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:48 dist-test-slave-2dw5 krb5kdc[10577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:48 dist-test-slave-2dw5 krb5kdc[10577](info): set up 2 sockets
Sep 26 13:09:48 dist-test-slave-2dw5 krb5kdc[10577](info): commencing operation
krb5kdc: starting...
W20260926 13:09:49.603633 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:49 dist-test-slave-2dw5 krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:09:49.722651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:49.803607 10590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:49.803833 10590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:49.803862 10590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:49.805080 10590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:09:49.805128 10590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:49.805147 10590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:49.805163 10590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:09:49.805179 10590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:09:49.806737 10590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:49.807050 10590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:49.807281 10590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:49.812191 10596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:49.812242 10590 server_base.cc:1034] running on GCE node
W20260926 13:09:49.812204 10595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:49.812186 10598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:49.812736 10590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:49.813102 10590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:49.814267 10590 hybrid_clock.cc:648] HybridClock initialized: now 1790428189814253 us; error 36 us; skew 500 ppm
Sep 26 13:09:49 dist-test-slave-2dw5 krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:49.823200 10590 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:09:49.823699 10590 webserver.cc:460] Webserver started at http://127.0.98.62:39979/ using document root <none> and password file <none>
I20260926 13:09:49.823928 10590 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:49.823978 10590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:49.824100 10590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:49.825181 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "135c27a32489413ab37863e8e8803222"
format_stamp: "Formatted at 2026-09-26 13:09:49 on dist-test-slave-2dw5"
server_key: "be430f8ab7e1544e3711f78ae5f15a82"
server_key_iv: "1f52e83e07ca302c1a9a69f08b593c83"
server_key_version: "encryptionkey@0"
I20260926 13:09:49.825517 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "135c27a32489413ab37863e8e8803222"
format_stamp: "Formatted at 2026-09-26 13:09:49 on dist-test-slave-2dw5"
server_key: "be430f8ab7e1544e3711f78ae5f15a82"
server_key_iv: "1f52e83e07ca302c1a9a69f08b593c83"
server_key_version: "encryptionkey@0"
I20260926 13:09:49.826921 10590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:09:49.827855 10605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:49.828097 10590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:49.828213 10590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "135c27a32489413ab37863e8e8803222"
format_stamp: "Formatted at 2026-09-26 13:09:49 on dist-test-slave-2dw5"
server_key: "be430f8ab7e1544e3711f78ae5f15a82"
server_key_iv: "1f52e83e07ca302c1a9a69f08b593c83"
server_key_version: "encryptionkey@0"
I20260926 13:09:49.828290 10590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:49.839991 10590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:49.842134 10590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:49.842275 10590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:49.846298 10590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44621
I20260926 13:09:49.846334 10657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44621 every 8 connection(s)
I20260926 13:09:49.846665 10590 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:09:49.847644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 10590
I20260926 13:09:49.847653 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:49.847733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:09:49.847914 392 external_mini_cluster.cc:1428] Setting key 946925a09dcb7e641d3bdda0cfdb70a8
I20260926 13:09:49.850950 10658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222: Bootstrap starting.
I20260926 13:09:49.851643 10658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:49.851939 10658 log.cc:824] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:49.852600 10658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222: No bootstrap required, opened a new log
Sep 26 13:09:49 dist-test-slave-2dw5 krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:49.854823 10658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135c27a32489413ab37863e8e8803222" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44621 } }
I20260926 13:09:49.855010 10658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:49.855033 10658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135c27a32489413ab37863e8e8803222, State: Initialized, Role: FOLLOWER
I20260926 13:09:49.855144 10658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135c27a32489413ab37863e8e8803222" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44621 } }
I20260926 13:09:49.855225 10658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:09:49.855257 10658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:09:49.855304 10658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:49.855957 10658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135c27a32489413ab37863e8e8803222" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44621 } }
I20260926 13:09:49.856058 10658 leader_election.cc:302] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135c27a32489413ab37863e8e8803222; no voters:
I20260926 13:09:49.856233 10658 leader_election.cc:288] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:09:49.856292 10663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:49.856494 10658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:09:49.856492 10663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [term 1 LEADER]: Becoming Leader. State: Replica: 135c27a32489413ab37863e8e8803222, State: Running, Role: LEADER
I20260926 13:09:49.856701 10663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135c27a32489413ab37863e8e8803222" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44621 } }
I20260926 13:09:49.857115 10665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135c27a32489413ab37863e8e8803222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135c27a32489413ab37863e8e8803222" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44621 } } }
I20260926 13:09:49.857115 10663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135c27a32489413ab37863e8e8803222. Latest consensus state: current_term: 1 leader_uuid: "135c27a32489413ab37863e8e8803222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135c27a32489413ab37863e8e8803222" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44621 } } }
I20260926 13:09:49.857231 10665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:49.857257 10663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222 [sys.catalog]: This master's current role is: LEADER
I20260926 13:09:49.857425 10672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:09:49.857999 10672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:09:49.859145 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:49.848428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.0.98.62:44621)
0926 13:09:49.848598 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:49.848612 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:49.848638 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:09:49.849070 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:49.849077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:49.849122 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:49.849231 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:49.849993 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:49.850956 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:49.851995 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:49.852419 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:49.852496 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:49.855143 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:49.855147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:49.855148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:49.855160 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:49.857343 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:49.857821 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:49.857824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:49.857824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:49.857927 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:49.858161 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:49.858161 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:49.858162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:49.858468 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:49.858635 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:49.858765 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 13:09:49.858939 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260926 13:09:49.859848 10672 catalog_manager.cc:1347] Generated new cluster ID: 6f6d3b9a07e043faa067fb54ff853a95
I20260926 13:09:49.859939 10672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:09:49.865681 10672 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:09:49.866436 10672 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:09:49.878450 10672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222: Generated new TSK 0
I20260926 13:09:49.878685 10672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:09:49.914145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44621
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:49.997912 10684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:49.998145 10684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:49.998178 10684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:49.999583 10684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:49.999637 10684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:49.999675 10684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:09:50.001210 10684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:50.001459 10684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:50.001672 10684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:50.006695 10692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:50.006863 10690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:50.006798 10689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:50.007085 10684 server_base.cc:1034] running on GCE node
I20260926 13:09:50.007288 10684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:50.007586 10684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:50.008855 10684 hybrid_clock.cc:648] HybridClock initialized: now 1790428190008703 us; error 68 us; skew 500 ppm
Sep 26 13:09:50 dist-test-slave-2dw5 krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:50.011564 10684 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:09:50.012184 10684 webserver.cc:460] Webserver started at http://127.0.98.1:46619/ using document root <none> and password file <none>
I20260926 13:09:50.012413 10684 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:50.012461 10684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:50.012588 10684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:50.013691 10684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "73e9a9a0ecb149449dceee78ce669509"
format_stamp: "Formatted at 2026-09-26 13:09:50 on dist-test-slave-2dw5"
server_key: "29ac71c4ec01a147596107e788c16d2f"
server_key_iv: "916a898965cf8a76565a2766cd17157f"
server_key_version: "encryptionkey@0"
I20260926 13:09:50.014040 10684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "73e9a9a0ecb149449dceee78ce669509"
format_stamp: "Formatted at 2026-09-26 13:09:50 on dist-test-slave-2dw5"
server_key: "29ac71c4ec01a147596107e788c16d2f"
server_key_iv: "916a898965cf8a76565a2766cd17157f"
server_key_version: "encryptionkey@0"
I20260926 13:09:50.015468 10684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:50.016435 10699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:50.016672 10684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:50.016763 10684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "73e9a9a0ecb149449dceee78ce669509"
format_stamp: "Formatted at 2026-09-26 13:09:50 on dist-test-slave-2dw5"
server_key: "29ac71c4ec01a147596107e788c16d2f"
server_key_iv: "916a898965cf8a76565a2766cd17157f"
server_key_version: "encryptionkey@0"
I20260926 13:09:50.016901 10684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:50.030427 10684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:50.032593 10684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:50.032732 10684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:50.032951 10684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:50.033226 10684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:50.033257 10684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:50.033284 10684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:50.033308 10684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:50.038774 10684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37729
I20260926 13:09:50.038955 10812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37729 every 8 connection(s)
I20260926 13:09:50.039160 10684 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:09:50.039527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 10684
I20260926 13:09:50.039620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:09:50.039794 392 external_mini_cluster.cc:1428] Setting key 03865beec62b8b6d734b2dcda2eb4705
Sep 26 13:09:50 dist-test-slave-2dw5 krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:09:50.049511 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.040023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49869 (local address 127.0.98.62:44621)
0926 13:09:50.040072 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:50.040075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:50.040211 (+ 136us) server_negotiation.cc:407] Connection header received
0926 13:09:50.040794 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:50.040795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:50.040806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:50.041018 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:50.042152 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.042678 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.044148 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.044366 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.044425 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.046810 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:50.046812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:50.046813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:50.046827 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:50.048208 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:50.048588 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:50.048589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:50.048589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:50.048656 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:50.048876 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:50.048877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:50.048878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:50.048990 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:50.049046 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:50.049316 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 13:09:50.049413 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:09:50.049767 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.039887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44621 (local address 127.0.98.1:49869)
0926 13:09:50.040077 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:50.040127 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:50.040700 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:50.041073 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:50.041077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:50.041731 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:50.042024 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.042031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.042776 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.042777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.044058 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.044061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.044453 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.044454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.044600 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.045152 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:50.045163 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:50.046718 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:50.048288 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:50.048296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:50.048299 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:50.048519 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:50.048711 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:50.048711 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:50.048712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:50.048820 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:50.049043 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:50.049051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:50.049197 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 13:09:50.049288 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 13:09:50.049461 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 13:09:50.050478 10813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44621
I20260926 13:09:50.050577 10813 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:50.050758 10813 heartbeater.cc:511] Master 127.0.98.62:44621 requested a full tablet report, sending...
I20260926 13:09:50.051389 10622 ts_manager.cc:194] Registered new tserver with Master: 73e9a9a0ecb149449dceee78ce669509 (127.0.98.1:37729)
I20260926 13:09:50.052057 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:09:50.065913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44621
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:50.146128 10818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:50.146355 10818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:50.146380 10818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:50.147688 10818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:50.147747 10818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:50.147778 10818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:09:50.149336 10818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:50.149585 10818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:50.149817 10818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:50.154950 10823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:50.155130 10818 server_base.cc:1034] running on GCE node
W20260926 13:09:50.154965 10826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:50.154950 10824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:50.155416 10818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:50.155663 10818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:50.156872 10818 hybrid_clock.cc:648] HybridClock initialized: now 1790428190156864 us; error 29 us; skew 500 ppm
Sep 26 13:09:50 dist-test-slave-2dw5 krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:50.159493 10818 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:09:50.160059 10818 webserver.cc:460] Webserver started at http://127.0.98.2:33761/ using document root <none> and password file <none>
I20260926 13:09:50.160281 10818 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:50.160339 10818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:50.160471 10818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:50.161553 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "e4fe84620b3b48d2878ed547689cf524"
format_stamp: "Formatted at 2026-09-26 13:09:50 on dist-test-slave-2dw5"
server_key: "d534194914821f0d867d8fb3df228a5e"
server_key_iv: "656ed184025e21ca4e3407fd97e627e9"
server_key_version: "encryptionkey@0"
I20260926 13:09:50.161908 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e4fe84620b3b48d2878ed547689cf524"
format_stamp: "Formatted at 2026-09-26 13:09:50 on dist-test-slave-2dw5"
server_key: "d534194914821f0d867d8fb3df228a5e"
server_key_iv: "656ed184025e21ca4e3407fd97e627e9"
server_key_version: "encryptionkey@0"
I20260926 13:09:50.163242 10818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:09:50.164148 10833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:50.164348 10818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:50.164434 10818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "e4fe84620b3b48d2878ed547689cf524"
format_stamp: "Formatted at 2026-09-26 13:09:50 on dist-test-slave-2dw5"
server_key: "d534194914821f0d867d8fb3df228a5e"
server_key_iv: "656ed184025e21ca4e3407fd97e627e9"
server_key_version: "encryptionkey@0"
I20260926 13:09:50.164497 10818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:50.177245 10818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:50.179550 10818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:50.179699 10818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:50.179940 10818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:50.180230 10818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:50.180263 10818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:50.180292 10818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:50.180316 10818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:50.185931 10818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43651
I20260926 13:09:50.186098 10946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43651 every 8 connection(s)
I20260926 13:09:50.186328 10818 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:09:50.190928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 10818
I20260926 13:09:50.191028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:09:50.191226 392 external_mini_cluster.cc:1428] Setting key ff1e33633ea83527ac57a599f508a074
Sep 26 13:09:50 dist-test-slave-2dw5 krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:50.196749 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.187105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35601 (local address 127.0.98.62:44621)
0926 13:09:50.187159 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:50.187161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:50.187398 (+ 237us) server_negotiation.cc:407] Connection header received
0926 13:09:50.187957 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:50.187959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:50.187970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:50.188033 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:50.189093 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.189589 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.190730 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.191045 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.191119 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.193666 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:50.193668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:50.193668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:50.193677 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:50.195208 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:50.195753 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:50.195754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:50.195754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:50.195807 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:50.196092 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:50.196093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:50.196093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:50.196205 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:50.196254 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:50.196610 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 13:09:50.196679 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 13:09:50.197036 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.187076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44621 (local address 127.0.98.2:35601)
0926 13:09:50.187239 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:50.187283 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:50.187776 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:50.188162 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:50.188166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:50.188776 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:50.188985 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.188990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.189687 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.189688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.190623 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.190626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.191131 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.191132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.191286 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.191881 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:50.191891 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:50.193556 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:50.195326 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:50.195338 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:50.195341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:50.195673 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:50.195871 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:50.195871 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:50.195872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:50.196024 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:50.196264 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:50.196271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:50.196474 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 13:09:50.196599 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 13:09:50.196785 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260926 13:09:50.197577 10947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44621
I20260926 13:09:50.197679 10947 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:50.197872 10947 heartbeater.cc:511] Master 127.0.98.62:44621 requested a full tablet report, sending...
I20260926 13:09:50.198274 10622 ts_manager.cc:194] Registered new tserver with Master: e4fe84620b3b48d2878ed547689cf524 (127.0.98.2:43651)
I20260926 13:09:50.198807 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35601
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:09:50.218174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44621
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:09:50.298971 10952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:09:50.299194 10952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:09:50.299226 10952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:09:50.300457 10952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:09:50.300511 10952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:09:50.300540 10952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:09:50.302076 10952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:09:50.302317 10952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:09:50.302551 10952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:09:50.307693 10958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:50.307677 10957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:09:50.307732 10960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:09:50.307717 10952 server_base.cc:1034] running on GCE node
I20260926 13:09:50.308184 10952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:09:50.308436 10952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:09:50.309598 10952 hybrid_clock.cc:648] HybridClock initialized: now 1790428190309579 us; error 36 us; skew 500 ppm
Sep 26 13:09:50 dist-test-slave-2dw5 krb5kdc[10577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:09:50.312419 10952 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:09:50.312968 10952 webserver.cc:460] Webserver started at http://127.0.98.3:44721/ using document root <none> and password file <none>
I20260926 13:09:50.313184 10952 fs_manager.cc:359] Metadata directory not provided
I20260926 13:09:50.313235 10952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:09:50.313369 10952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:09:50.314507 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "c96c000dd22c454f9ea413d97a151db0"
format_stamp: "Formatted at 2026-09-26 13:09:50 on dist-test-slave-2dw5"
server_key: "a85d2042d6876e5e74e56cc5ed83aa60"
server_key_iv: "658725098017c70cbc0895ba0b0b2057"
server_key_version: "encryptionkey@0"
I20260926 13:09:50.314924 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c96c000dd22c454f9ea413d97a151db0"
format_stamp: "Formatted at 2026-09-26 13:09:50 on dist-test-slave-2dw5"
server_key: "a85d2042d6876e5e74e56cc5ed83aa60"
server_key_iv: "658725098017c70cbc0895ba0b0b2057"
server_key_version: "encryptionkey@0"
I20260926 13:09:50.316313 10952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:50.317091 10967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:50.317368 10952 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:09:50.317453 10952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "c96c000dd22c454f9ea413d97a151db0"
format_stamp: "Formatted at 2026-09-26 13:09:50 on dist-test-slave-2dw5"
server_key: "a85d2042d6876e5e74e56cc5ed83aa60"
server_key_iv: "658725098017c70cbc0895ba0b0b2057"
server_key_version: "encryptionkey@0"
I20260926 13:09:50.317524 10952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:09:50.329919 10952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:09:50.332227 10952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:09:50.332368 10952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:09:50.332599 10952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:09:50.332906 10952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:09:50.332940 10952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:50.332968 10952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:09:50.332991 10952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:09:50.338614 10952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44657
I20260926 13:09:50.338652 11080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44657 every 8 connection(s)
I20260926 13:09:50.338968 10952 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:09:50.343008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 10952
I20260926 13:09:50.343123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:09:50.343309 392 external_mini_cluster.cc:1428] Setting key 82770a68fcad44745ecf46efc7a9804a
Sep 26 13:09:50 dist-test-slave-2dw5 krb5kdc[10577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:09:50.348018 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.339626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52755 (local address 127.0.98.62:44621)
0926 13:09:50.339679 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:50.339681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:50.339854 (+ 173us) server_negotiation.cc:407] Connection header received
0926 13:09:50.340325 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:50.340327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:50.340338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:50.340387 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:50.341285 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.341782 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.342748 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.343040 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.343104 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.345593 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:50.345594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:50.345594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:50.345600 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:50.346708 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:50.347124 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:50.347125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:50.347125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:50.347175 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:50.347413 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:50.347414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:50.347414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:50.347546 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:50.347609 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:50.347828 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 13:09:50.347955 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 13:09:50.348112 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.339566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44621 (local address 127.0.98.3:52755)
0926 13:09:50.339710 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:50.339750 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:50.340236 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:50.340455 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:50.340459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:50.341006 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:09:50.341187 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.341192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.341861 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.341862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.342665 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.342667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.343142 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.343143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.343320 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.344026 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:09:50.344035 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:09:50.345510 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:09:50.346784 (+ 1274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:50.346791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:50.346793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:50.347061 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:50.347223 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:09:50.347224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:09:50.347224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:09:50.347339 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:09:50.347606 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:09:50.347610 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:09:50.347725 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 13:09:50.347817 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:09:50.347936 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260926 13:09:50.348505 11081 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44621
I20260926 13:09:50.348591 11081 heartbeater.cc:464] Registering TS with master...
I20260926 13:09:50.348752 11081 heartbeater.cc:511] Master 127.0.98.62:44621 requested a full tablet report, sending...
I20260926 13:09:50.349196 10622 ts_manager.cc:194] Registered new tserver with Master: c96c000dd22c454f9ea413d97a151db0 (127.0.98.3:44657)
I20260926 13:09:50.349570 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52755
I20260926 13:09:50.355039 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:09:50.361889 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.356315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.0.98.62:44621)
0926 13:09:50.356358 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:50.356359 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:50.356472 (+ 113us) server_negotiation.cc:407] Connection header received
0926 13:09:50.356511 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:50.356512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:50.356520 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:50.356564 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:09:50.357157 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.357713 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.358445 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.358622 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.358671 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.359818 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:09:50.359819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:09:50.359819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:09:50.359825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:09:50.360961 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:50.361269 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:50.361270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:50.361271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:50.361328 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:09:50.361467 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:09:50.361468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:09:50.361468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:09:50.361606 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:09:50.361678 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:50.361758 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 13:09:50.361827 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:09:50.363644 10622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:09:50.365238 10622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:09:50.371536 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.367395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.0.98.1:37729)
0926 13:09:50.367581 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:50.367585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:50.367692 (+ 107us) server_negotiation.cc:407] Connection header received
0926 13:09:50.367787 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:50.367789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:50.367805 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:50.367965 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:50.368127 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.369302 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.370377 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.371138 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.371209 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.371267 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:50.371410 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 13:09:50.371453 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260926 13:09:50.371551 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.367222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43651 (local address 127.0.0.1:59258)
0926 13:09:50.367916 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:50.367925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:50.367965 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:50.368205 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:50.368206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:50.368209 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:50.368281 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.368284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.369185 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.369186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.370365 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.370368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.371301 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.371302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.371372 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.371376 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:09:50.371423 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 13:09:50.371465 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":12,"thread_start_us":55,"threads_started":1}
I20260926 13:09:50.371558 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.367220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37729 (local address 127.0.0.1:43426)
0926 13:09:50.367597 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:50.367636 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:50.367710 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:50.367970 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:50.367973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:50.367975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:50.368050 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.368052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.369421 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.369423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.370305 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.370307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.371227 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.371228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.371306 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.371320 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 13:09:50.371389 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:09:50.371443 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":71,"threads_started":1}
I20260926 13:09:50.371865 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.367667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.0.98.3:44657)
0926 13:09:50.367832 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:50.367836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:50.367846 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:09:50.367915 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:50.367917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:50.367932 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:50.368125 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:50.368355 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.369406 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.370611 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.371166 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.371231 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.371287 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:50.371508 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 13:09:50.371548 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260926 13:09:50.371878 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.367403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.0.98.2:43651)
0926 13:09:50.367658 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:50.367661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:50.368013 (+ 352us) server_negotiation.cc:407] Connection header received
0926 13:09:50.368056 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:50.368058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:50.368074 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:50.368138 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:50.368357 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.369100 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.370443 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.371140 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.371209 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.371267 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:50.371754 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 13:09:50.371797 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":89,"threads_started":1}
I20260926 13:09:50.372359 10747 tablet_service.cc:1467] Processing CreateTablet for tablet 401ea193fe654bb588f2e312307d67dc (DEFAULT_TABLE table=test-table [id=bbdd5c7ca8db4c5681830ceb80c70737]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:50.372517 10881 tablet_service.cc:1467] Processing CreateTablet for tablet 401ea193fe654bb588f2e312307d67dc (DEFAULT_TABLE table=test-table [id=bbdd5c7ca8db4c5681830ceb80c70737]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:50.372659 10747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401ea193fe654bb588f2e312307d67dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:50.372830 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401ea193fe654bb588f2e312307d67dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:50.375267 11096 tablet_bootstrap.cc:492] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509: Bootstrap starting.
I20260926 13:09:50.371767 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.367222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44657 (local address 127.0.0.1:47452)
0926 13:09:50.367802 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:50.367811 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:50.367845 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:50.368205 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:50.368205 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:50.368207 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:50.368281 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.368283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.369495 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.369496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.370543 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.370545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.371317 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.371318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.371377 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.371380 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:50.371433 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 13:09:50.371468 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":5,"thread_start_us":58,"threads_started":1}
I20260926 13:09:50.375936 11096 tablet_bootstrap.cc:654] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:50.376072 11015 tablet_service.cc:1467] Processing CreateTablet for tablet 401ea193fe654bb588f2e312307d67dc (DEFAULT_TABLE table=test-table [id=bbdd5c7ca8db4c5681830ceb80c70737]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:09:50.376427 11015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401ea193fe654bb588f2e312307d67dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:09:50.376461 11096 log.cc:824] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:50.376797 11095 tablet_bootstrap.cc:492] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524: Bootstrap starting.
I20260926 13:09:50.377141 11096 tablet_bootstrap.cc:492] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509: No bootstrap required, opened a new log
I20260926 13:09:50.377203 11096 ts_tablet_manager.cc:1397] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:09:50.377403 11095 tablet_bootstrap.cc:654] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:50.377929 11095 log.cc:824] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:50.378744 11095 tablet_bootstrap.cc:492] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524: No bootstrap required, opened a new log
I20260926 13:09:50.378823 11095 ts_tablet_manager.cc:1397] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:09:50.379000 11096 raft_consensus.cc:348] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe84620b3b48d2878ed547689cf524" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "73e9a9a0ecb149449dceee78ce669509" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37729 } } peers { permanent_uuid: "c96c000dd22c454f9ea413d97a151db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44657 } }
I20260926 13:09:50.379139 11096 raft_consensus.cc:374] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:50.379171 11096 raft_consensus.cc:729] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e9a9a0ecb149449dceee78ce669509, State: Initialized, Role: FOLLOWER
I20260926 13:09:50.379287 11096 consensus_queue.cc:260] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe84620b3b48d2878ed547689cf524" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "73e9a9a0ecb149449dceee78ce669509" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37729 } } peers { permanent_uuid: "c96c000dd22c454f9ea413d97a151db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44657 } }
I20260926 13:09:50.379575 10813 heartbeater.cc:503] Master 127.0.98.62:44621 was elected leader, sending a full tablet report...
I20260926 13:09:50.379596 11096 ts_tablet_manager.cc:1428] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 13:09:50.380941 11095 raft_consensus.cc:348] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe84620b3b48d2878ed547689cf524" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "73e9a9a0ecb149449dceee78ce669509" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37729 } } peers { permanent_uuid: "c96c000dd22c454f9ea413d97a151db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44657 } }
I20260926 13:09:50.381078 11095 raft_consensus.cc:374] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:50.381098 11095 raft_consensus.cc:729] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fe84620b3b48d2878ed547689cf524, State: Initialized, Role: FOLLOWER
I20260926 13:09:50.381201 11095 consensus_queue.cc:260] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe84620b3b48d2878ed547689cf524" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "73e9a9a0ecb149449dceee78ce669509" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37729 } } peers { permanent_uuid: "c96c000dd22c454f9ea413d97a151db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44657 } }
I20260926 13:09:50.381435 11095 ts_tablet_manager.cc:1428] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:09:50.381520 10947 heartbeater.cc:503] Master 127.0.98.62:44621 was elected leader, sending a full tablet report...
I20260926 13:09:50.381621 11100 tablet_bootstrap.cc:492] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0: Bootstrap starting.
I20260926 13:09:50.382212 11100 tablet_bootstrap.cc:654] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0: Neither blocks nor log segments found. Creating new log.
I20260926 13:09:50.382653 11100 log.cc:824] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0: Log is configured to *not* fsync() on all Append() calls
I20260926 13:09:50.383343 11100 tablet_bootstrap.cc:492] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0: No bootstrap required, opened a new log
I20260926 13:09:50.383402 11100 ts_tablet_manager.cc:1397] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 13:09:50.384733 11100 raft_consensus.cc:348] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe84620b3b48d2878ed547689cf524" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "73e9a9a0ecb149449dceee78ce669509" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37729 } } peers { permanent_uuid: "c96c000dd22c454f9ea413d97a151db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44657 } }
I20260926 13:09:50.384882 11100 raft_consensus.cc:374] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:09:50.384904 11100 raft_consensus.cc:729] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96c000dd22c454f9ea413d97a151db0, State: Initialized, Role: FOLLOWER
I20260926 13:09:50.385004 11100 consensus_queue.cc:260] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe84620b3b48d2878ed547689cf524" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "73e9a9a0ecb149449dceee78ce669509" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37729 } } peers { permanent_uuid: "c96c000dd22c454f9ea413d97a151db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44657 } }
I20260926 13:09:50.385246 11100 ts_tablet_manager.cc:1428] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:09:50.385303 11081 heartbeater.cc:503] Master 127.0.98.62:44621 was elected leader, sending a full tablet report...
W20260926 13:09:50.436970 10948 tablet.cc:2369] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:09:50.472942 11099 raft_consensus.cc:482] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:09:50.473076 11099 raft_consensus.cc:504] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe84620b3b48d2878ed547689cf524" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "73e9a9a0ecb149449dceee78ce669509" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37729 } } peers { permanent_uuid: "c96c000dd22c454f9ea413d97a151db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44657 } }
I20260926 13:09:50.473417 11099 leader_election.cc:288] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4fe84620b3b48d2878ed547689cf524 (127.0.98.2:43651), c96c000dd22c454f9ea413d97a151db0 (127.0.98.3:44657)
I20260926 13:09:50.476290 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.473705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36413 (local address 127.0.98.3:44657)
0926 13:09:50.473759 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:50.473761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:50.473897 (+ 136us) server_negotiation.cc:407] Connection header received
0926 13:09:50.473927 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:50.473928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:50.473940 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:50.473990 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:50.474169 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.474618 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.475613 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.475976 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.476038 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.476047 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:50.476189 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 13:09:50.476223 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:09:50.476303 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.473729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43377 (local address 127.0.98.2:43651)
0926 13:09:50.473876 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:50.473877 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:50.473888 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:09:50.473914 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:50.473915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:50.473925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:50.473991 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:09:50.474129 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.474604 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.475569 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.475955 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.476015 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.476027 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:50.476178 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 13:09:50.476217 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260926 13:09:50.476300 10815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.473625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43651 (local address 127.0.98.1:43377)
0926 13:09:50.473688 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:50.473699 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:50.473745 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:50.473999 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:50.474000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:50.474001 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:50.474064 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.474065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.474687 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.474688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.475475 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.475478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.476033 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.476034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.476108 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.476111 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:50.476151 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 13:09:50.476192 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 13:09:50.476392 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.473645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44657 (local address 127.0.98.1:36413)
0926 13:09:50.473828 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:50.473836 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:09:50.473883 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:09:50.474021 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:09:50.474022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:09:50.474024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:09:50.474094 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.474096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.474699 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.474700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.475524 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:09:50.475527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.476056 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.476057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:09:50.476119 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.476122 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:09:50.476178 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 13:09:50.476217 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":120,"threads_started":1}
I20260926 13:09:50.476665 10901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401ea193fe654bb588f2e312307d67dc" candidate_uuid: "73e9a9a0ecb149449dceee78ce669509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fe84620b3b48d2878ed547689cf524" is_pre_election: true
I20260926 13:09:50.476748 11035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401ea193fe654bb588f2e312307d67dc" candidate_uuid: "73e9a9a0ecb149449dceee78ce669509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96c000dd22c454f9ea413d97a151db0" is_pre_election: true
I20260926 13:09:50.476840 10901 raft_consensus.cc:2454] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73e9a9a0ecb149449dceee78ce669509 in term 0.
I20260926 13:09:50.476922 11035 raft_consensus.cc:2454] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73e9a9a0ecb149449dceee78ce669509 in term 0.
I20260926 13:09:50.477069 10700 leader_election.cc:302] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73e9a9a0ecb149449dceee78ce669509, e4fe84620b3b48d2878ed547689cf524; no voters:
I20260926 13:09:50.477176 11099 raft_consensus.cc:2781] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:09:50.477244 11099 raft_consensus.cc:482] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:09:50.477288 11099 raft_consensus.cc:3037] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:50.478014 11099 raft_consensus.cc:504] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe84620b3b48d2878ed547689cf524" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "73e9a9a0ecb149449dceee78ce669509" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37729 } } peers { permanent_uuid: "c96c000dd22c454f9ea413d97a151db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44657 } }
I20260926 13:09:50.478149 11099 leader_election.cc:288] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [CANDIDATE]: Term 1 election: Requested vote from peers e4fe84620b3b48d2878ed547689cf524 (127.0.98.2:43651), c96c000dd22c454f9ea413d97a151db0 (127.0.98.3:44657)
I20260926 13:09:50.478360 10901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401ea193fe654bb588f2e312307d67dc" candidate_uuid: "73e9a9a0ecb149449dceee78ce669509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fe84620b3b48d2878ed547689cf524"
I20260926 13:09:50.478456 10901 raft_consensus.cc:3037] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:50.478590 11035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "401ea193fe654bb588f2e312307d67dc" candidate_uuid: "73e9a9a0ecb149449dceee78ce669509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96c000dd22c454f9ea413d97a151db0"
I20260926 13:09:50.478672 11035 raft_consensus.cc:3037] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:09:50.479286 10901 raft_consensus.cc:2454] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73e9a9a0ecb149449dceee78ce669509 in term 1.
I20260926 13:09:50.479475 11035 raft_consensus.cc:2454] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73e9a9a0ecb149449dceee78ce669509 in term 1.
I20260926 13:09:50.479525 10700 leader_election.cc:302] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73e9a9a0ecb149449dceee78ce669509, e4fe84620b3b48d2878ed547689cf524; no voters:
I20260926 13:09:50.479619 11099 raft_consensus.cc:2781] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:09:50.479773 11099 raft_consensus.cc:686] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [term 1 LEADER]: Becoming Leader. State: Replica: 73e9a9a0ecb149449dceee78ce669509, State: Running, Role: LEADER
I20260926 13:09:50.479862 11099 consensus_queue.cc:237] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe84620b3b48d2878ed547689cf524" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } } peers { permanent_uuid: "73e9a9a0ecb149449dceee78ce669509" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37729 } } peers { permanent_uuid: "c96c000dd22c454f9ea413d97a151db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44657 } }
I20260926 13:09:50.480767 10622 catalog_manager.cc:5495] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73e9a9a0ecb149449dceee78ce669509 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "73e9a9a0ecb149449dceee78ce669509" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fe84620b3b48d2878ed547689cf524" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73e9a9a0ecb149449dceee78ce669509" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c96c000dd22c454f9ea413d97a151db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44657 } health_report { overall_health: UNKNOWN } } }
I20260926 13:09:50.486959 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.484664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.0.98.1:37729)
0926 13:09:50.484720 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:09:50.484722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:09:50.484745 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:09:50.484778 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:09:50.484779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:09:50.484798 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:09:50.484856 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:09:50.485000 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.485482 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.486205 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:09:50.486413 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:09:50.486465 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:09:50.486621 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:09:50.486726 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:09:50.486798 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:09:50.486817 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 13:09:50.486864 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:09:50.488869 11035 raft_consensus.cc:1261] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0 [term 1 FOLLOWER]: Refusing update from remote peer 73e9a9a0ecb149449dceee78ce669509: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:50.488865 10901 raft_consensus.cc:1261] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524 [term 1 FOLLOWER]: Refusing update from remote peer 73e9a9a0ecb149449dceee78ce669509: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:09:50.489219 11099 consensus_queue.cc:1035] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4fe84620b3b48d2878ed547689cf524" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:50.489382 11105 consensus_queue.cc:1035] T 401ea193fe654bb588f2e312307d67dc P 73e9a9a0ecb149449dceee78ce669509 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c96c000dd22c454f9ea413d97a151db0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:09:50.491791 11108 mvcc.cc:205] Tried to move back new op lower bound from 7333593868239110144 to 7333593868206092288. Current Snapshot: MvccSnapshot[applied={T|T < 7333593868239110144}]
I20260926 13:09:50.492425 11109 mvcc.cc:205] Tried to move back new op lower bound from 7333593868239110144 to 7333593868206092288. Current Snapshot: MvccSnapshot[applied={T|T < 7333593868239110144}]
I20260926 13:09:50.495133 10622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:50.495246 10622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:09:50.496083 10622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 135c27a32489413ab37863e8e8803222: Sending DeleteTablet for 3 replicas of tablet 401ea193fe654bb588f2e312307d67dc
I20260926 13:09:50.496492 10747 tablet_service.cc:1514] Processing DeleteTablet for tablet 401ea193fe654bb588f2e312307d67dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43426
I20260926 13:09:50.496505 10881 tablet_service.cc:1514] Processing DeleteTablet for tablet 401ea193fe654bb588f2e312307d67dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59258
I20260926 13:09:50.496755 11015 tablet_service.cc:1514] Processing DeleteTablet for tablet 401ea193fe654bb588f2e312307d67dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:09:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47452
I20260926 13:09:50.496986 11120 tablet_replica.cc:330] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0: stopping tablet replica
I20260926 13:09:50.497100 11120 raft_consensus.cc:2229] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:50.497181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 10684
I20260926 13:09:50.497217 11120 raft_consensus.cc:2258] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:50.497807 11119 tablet_replica.cc:330] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524: stopping tablet replica
I20260926 13:09:50.497932 11119 raft_consensus.cc:2229] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:09:50.497932 11120 ts_tablet_manager.cc:1905] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:50.498028 11119 raft_consensus.cc:2258] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:09:50.499452 11119 ts_tablet_manager.cc:1905] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:09:50.499759 11120 ts_tablet_manager.cc:1918] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:50.499878 11120 log.cc:1196] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/401ea193fe654bb588f2e312307d67dc
I20260926 13:09:50.500203 11120 ts_tablet_manager.cc:1939] T 401ea193fe654bb588f2e312307d67dc P c96c000dd22c454f9ea413d97a151db0: Deleting consensus metadata
I20260926 13:09:50.500752 10608 catalog_manager.cc:4841] TS c96c000dd22c454f9ea413d97a151db0 (127.0.98.3:44657): tablet 401ea193fe654bb588f2e312307d67dc (table test-table [id=bbdd5c7ca8db4c5681830ceb80c70737]) successfully deleted
I20260926 13:09:50.501142 11119 ts_tablet_manager.cc:1918] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:09:50.501220 11119 log.cc:1196] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/401ea193fe654bb588f2e312307d67dc
I20260926 13:09:50.501480 11119 ts_tablet_manager.cc:1939] T 401ea193fe654bb588f2e312307d67dc P e4fe84620b3b48d2878ed547689cf524: Deleting consensus metadata
I20260926 13:09:50.502094 10606 catalog_manager.cc:4841] TS e4fe84620b3b48d2878ed547689cf524 (127.0.98.2:43651): tablet 401ea193fe654bb588f2e312307d67dc (table test-table [id=bbdd5c7ca8db4c5681830ceb80c70737]) successfully deleted
W20260926 13:09:50.502434 10607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37729 (error 108)
I20260926 13:09:50.502668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 10818
I20260926 13:09:50.502884 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:09:50.502757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37729 (local address 127.0.0.1:43438)
0926 13:09:50.502801 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 13:09:50.502831 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 13:09:50.502982 10607 catalog_manager.cc:4584] TS 73e9a9a0ecb149449dceee78ce669509 (127.0.98.1:37729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 401ea193fe654bb588f2e312307d67dc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37729: connect: Connection refused (error 111)
I20260926 13:09:50.507799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 10952
I20260926 13:09:50.511698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 10590
2026-09-26T13:09:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1875 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:09:50 dist-test-slave-2dw5 krb5kdc[11124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:09:50 dist-test-slave-2dw5 krb5kdc[11124](info): set up 2 sockets
Sep 26 13:09:50 dist-test-slave-2dw5 krb5kdc[11124](info): commencing operation
krb5kdc: starting...
W20260926 13:09:51.479581 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:09:51 dist-test-slave-2dw5 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:09:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 13:09:51.645857 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasky34iiX/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/postgres -l logfile start
2026-09-26 13:09:54.238 UTC [11154] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 13:09:54.238 UTC [11154] LOG: listening on IPv4 address "127.0.98.20", port 33881
2026-09-26 13:09:54.242 UTC [11154] LOG: listening on Unix socket "/tmp/.s.PGSQL.33881"
2026-09-26 13:09:54.270 UTC [11157] LOG: database system was shut down at 2026-09-26 13:09:52 UTC
2026-09-26 13:09:54.274 UTC [11154] LOG: database system is ready to accept connections
I20260926 13:09:55.147516 392 mini_postgres.cc:96] Postgres bound to 33881
2026-09-26 13:09:55.153 UTC [11165] FATAL: database "slave" does not exist
127.0.98.20:33881 - accepting connections
I20260926 13:09:55.154206 392 mini_ranger.cc:161] Starting Ranger...
I20260926 13:09:55.168612 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 13:09:55.717515 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 13:09:55.717614 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-admin
2026-09-26 13:09:56,031 [I] DB FLAVOR :POSTGRES
2026-09-26 13:09:56,032 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:09:56,032 [I] Checking connection..
2026-09-26 13:09:56,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 13:09:56,320 [I] Checking connection passed.
2026-09-26 13:09:56,321 [I] --------- Verifying version history table ---------
2026-09-26 13:09:56,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:09:56,597 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:09:56,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:09:56,857 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:09:56,857 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 13:09:56,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 13:09:57.117 UTC [11229] WARNING: there is no transaction in progress
2026-09-26 13:09:57,123 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 13:09:57,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:09:57,386 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 13:09:57,386 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:09:57,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 13:09:57,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 13:09:57,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2dw5.c.gcp-upstream.internal','N') ;"
2026-09-26 13:09:58,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:09:58,368 [I] Table x_portal_user does not exist in database ranger
2026-09-26 13:09:58,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:09:58,632 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 13:09:58,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:09:58,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:09:59,126 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 13:09:59,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 13:10:00.162 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.181 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.199 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.219 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.241 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.266 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.340 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.348 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.365 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.378 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.390 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.407 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.418 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.426 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.434 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:00.442 UTC [11337] WARNING: there is no transaction in progress
2026-09-26 13:10:01,225 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 13:10:01,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 13:10:01,462 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 13:10:01,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:10:01,729 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 13:10:01,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:10:02,002 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 13:10:02,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:10:02,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:10:02,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2dw5.c.gcp-upstream.internal';"
2026-09-26 13:10:02,774 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 13:10:02,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 13:10:03,026 [I] DB_PATCHES have already been applied
I20260926 13:10:03.032802 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-admin:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasky34iiX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 13:10:03.032878 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 13:10:03.035861 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:50025
Sep 26, 2026 1:10:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:10:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:10:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:10:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:10:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:10:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:10:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 13:10:04.067243 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.913s user 0.000s sys 0.009s
Sep 26, 2026 1:10:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:10:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 1:10:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:10:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:10:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:10:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 13:10:25.410370 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 13:10:25 dist-test-slave-2dw5 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428225, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 13:10:25.496542 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 13:10:25.506347 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 13:10:26.503026 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 13:10:26.503132 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-kms
I20260926 13:10:26.503149 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:33881
2026-09-26 13:10:26,714 [I] DB FLAVOR :POSTGRES
2026-09-26 13:10:26,714 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:10:26,714 [I] Checking connection
2026-09-26 13:10:26,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 13:10:26,989 [I] connection success
2026-09-26 13:10:26,989 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 13:10:26,989 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 13:10:26,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 13:10:27,262 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 13:10:27,262 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:10:27,263 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 13:10:27,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33881/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 13:10:27,541 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 13:10:28.026950 392 mini_ranger_kms.cc:310] Created kms service
I20260926 13:10:28.229130 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 13:10:28.278007 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 13:10:28 dist-test-slave-2dw5 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428225, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:10:28.685393 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 13:10:28.685509 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-kms:/tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 13:10:28.685530 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 13:10:28.688598 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11517
I20260926 13:10:28.688722 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54611
Sep 26, 2026 1:10:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:10:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:10:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:10:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:10:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:10:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 13:10:30.746665 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.250s user 0.010s sys 0.181s
I20260926 13:10:30.746774 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 13:10:30.746882 392 mini_ranger_kms.cc:401] 127.0.98.20:54611/kms/v1/keys
Sep 26 13:10:30 dist-test-slave-2dw5 krb5kdc[11124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790428230, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 13:10:31 dist-test-slave-2dw5 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428230, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 13:10:31 dist-test-slave-2dw5 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428230, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 13:10:34 dist-test-slave-2dw5 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:10:34.158041 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41603
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41603
--ranger_config_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54611/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:10:34.241025 11568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:10:34.241258 11568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:10:34.241281 11568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:10:34.242502 11568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:10:34.242544 11568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:10:34.242558 11568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:10:34.242568 11568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:10:34.242578 11568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:10:34.244243 11568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54611/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41603
--ranger_config_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:10:34.244489 11568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:10:34.244704 11568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:10:34.250862 11576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:10:34.250880 11573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:10:34.251034 11568 server_base.cc:1034] running on GCE node
W20260926 13:10:34.250874 11574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:10:34.251602 11568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:10:34.252012 11568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:10:34.253186 11568 hybrid_clock.cc:648] HybridClock initialized: now 1790428234253187 us; error 52 us; skew 500 ppm
Sep 26 13:10:34 dist-test-slave-2dw5 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:10:34.269744 11568 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:10:34.270301 11568 webserver.cc:460] Webserver started at http://127.0.98.62:36723/ using document root <none> and password file <none>
I20260926 13:10:34.270612 11568 fs_manager.cc:359] Metadata directory not provided
I20260926 13:10:34.270656 11568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:10:34.270792 11568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:10:34 dist-test-slave-2dw5 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:10:34.333832 11568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "1a2cf64b712a4ba6be0ac483fb4a9154"
format_stamp: "Formatted at 2026-09-26 13:10:34 on dist-test-slave-2dw5"
server_key: "ca774078782b92b6c97be452bad5a96e"
server_key_iv: "4536799fae5f118588dfa0f6b58904be"
server_key_version: "kuduclusterkey@0"
I20260926 13:10:34.334298 11568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a2cf64b712a4ba6be0ac483fb4a9154"
format_stamp: "Formatted at 2026-09-26 13:10:34 on dist-test-slave-2dw5"
server_key: "ca774078782b92b6c97be452bad5a96e"
server_key_iv: "4536799fae5f118588dfa0f6b58904be"
server_key_version: "kuduclusterkey@0"
I20260926 13:10:34.335774 11568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:10:34.358150 11583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:10:34.358410 11568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:10:34.360337 11568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "1a2cf64b712a4ba6be0ac483fb4a9154"
format_stamp: "Formatted at 2026-09-26 13:10:34 on dist-test-slave-2dw5"
server_key: "ca774078782b92b6c97be452bad5a96e"
server_key_iv: "4536799fae5f118588dfa0f6b58904be"
server_key_version: "kuduclusterkey@0"
I20260926 13:10:34.360657 11568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:10:34.370062 11568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:10:34.371892 11568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:10:34.372026 11568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:10:34.376475 11568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41603
I20260926 13:10:34.376508 11635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41603 every 8 connection(s)
I20260926 13:10:34.376921 11568 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:10:34.377538 11636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:10:34.379817 11636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154: Bootstrap starting.
I20260926 13:10:34.380390 11636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154: Neither blocks nor log segments found. Creating new log.
I20260926 13:10:34.380623 11636 log.cc:824] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154: Log is configured to *not* fsync() on all Append() calls
I20260926 13:10:34.381244 11636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154: No bootstrap required, opened a new log
I20260926 13:10:34.383224 11636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2cf64b712a4ba6be0ac483fb4a9154" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41603 } }
I20260926 13:10:34.383363 11636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:10:34.383394 11636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2cf64b712a4ba6be0ac483fb4a9154, State: Initialized, Role: FOLLOWER
I20260926 13:10:34.383527 11636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2cf64b712a4ba6be0ac483fb4a9154" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41603 } }
I20260926 13:10:34.383599 11636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:10:34.383630 11636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:10:34.383674 11636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:10:34.384186 11636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2cf64b712a4ba6be0ac483fb4a9154" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41603 } }
I20260926 13:10:34.384276 11636 leader_election.cc:302] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a2cf64b712a4ba6be0ac483fb4a9154; no voters:
I20260926 13:10:34.384393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 11568
I20260926 13:10:34.384449 11636 leader_election.cc:288] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:10:34.384485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:10:34.384483 11639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:10:34.384601 11639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [term 1 LEADER]: Becoming Leader. State: Replica: 1a2cf64b712a4ba6be0ac483fb4a9154, State: Running, Role: LEADER
I20260926 13:10:34.384658 392 external_mini_cluster.cc:1428] Setting key e05d6a525201b89ce351ce7890ff8344
I20260926 13:10:34.384691 11636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:10:34.384670 11639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2cf64b712a4ba6be0ac483fb4a9154" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41603 } }
I20260926 13:10:34.384958 11639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a2cf64b712a4ba6be0ac483fb4a9154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2cf64b712a4ba6be0ac483fb4a9154" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41603 } } }
I20260926 13:10:34.385015 11639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [sys.catalog]: This master's current role is: LEADER
I20260926 13:10:34.385118 11639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a2cf64b712a4ba6be0ac483fb4a9154. Latest consensus state: current_term: 1 leader_uuid: "1a2cf64b712a4ba6be0ac483fb4a9154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2cf64b712a4ba6be0ac483fb4a9154" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41603 } } }
I20260926 13:10:34.385159 11639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154 [sys.catalog]: This master's current role is: LEADER
I20260926 13:10:34.386122 11636 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 13:10:34.390254 11644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:10:34.390925 11644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 13:10:34 dist-test-slave-2dw5 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:10:34.400681 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:34.385247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.0.98.62:41603)
0926 13:10:34.388607 (+ 3360us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:34.388624 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:34.388672 (+ 48us) server_negotiation.cc:407] Connection header received
0926 13:10:34.388867 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:34.388875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:34.388979 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:34.389147 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:10:34.389927 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:34.391522 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:34.392645 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:34.393275 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:34.393705 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:34.396382 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:10:34.396386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:10:34.396387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:10:34.396400 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:10:34.398444 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:10:34.398926 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:10:34.398929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:10:34.398930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:10:34.399034 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:10:34.399240 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:10:34.399241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:10:34.399241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:10:34.399596 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:10:34.400191 (+ 595us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:34.400283 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 13:10:34.400433 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3230,"thread_start_us":74,"threads_started":1}
I20260926 13:10:34.407605 11644 catalog_manager.cc:1347] Generated new cluster ID: 5a51c4f3b2f44187a5db5fcb0e12e017
I20260926 13:10:34.407680 11644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:10:34.419642 11644 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:10:34.420449 11644 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:10:34.428170 11644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154: Generated new TSK 0
I20260926 13:10:34.428503 11644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:10:35.237574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54611/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41603
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:10:35.360126 11681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:10:35.360427 11681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:10:35.360463 11681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:10:35.362268 11681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:10:35.362345 11681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:10:35.362390 11681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:10:35.364792 11681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54611/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:10:35.365074 11681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:10:35.365332 11681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:10:35.371114 11689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:10:35.371119 11686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:10:35.371433 11681 server_base.cc:1034] running on GCE node
W20260926 13:10:35.371244 11687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:10:35.371773 11681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:10:35.372017 11681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:10:35.373174 11681 hybrid_clock.cc:648] HybridClock initialized: now 1790428235373152 us; error 41 us; skew 500 ppm
Sep 26 13:10:35 dist-test-slave-2dw5 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:10:35.376037 11681 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:10:35.376619 11681 webserver.cc:460] Webserver started at http://127.0.98.1:35237/ using document root <none> and password file <none>
I20260926 13:10:35.376873 11681 fs_manager.cc:359] Metadata directory not provided
I20260926 13:10:35.376933 11681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:10:35.377108 11681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:10:35 dist-test-slave-2dw5 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:10:35.391947 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "97518660ff8c440f8169f75aba12e1a4"
format_stamp: "Formatted at 2026-09-26 13:10:35 on dist-test-slave-2dw5"
server_key: "378a21b2ee34f02bbb901fa9eede678b"
server_key_iv: "340c3c47bf88fb423c55137492e679d7"
server_key_version: "kuduclusterkey@0"
I20260926 13:10:35.392530 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "97518660ff8c440f8169f75aba12e1a4"
format_stamp: "Formatted at 2026-09-26 13:10:35 on dist-test-slave-2dw5"
server_key: "378a21b2ee34f02bbb901fa9eede678b"
server_key_iv: "340c3c47bf88fb423c55137492e679d7"
server_key_version: "kuduclusterkey@0"
I20260926 13:10:35.394399 11681 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:10:35.411199 11698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:10:35.411463 11681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:10:35.411554 11681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "97518660ff8c440f8169f75aba12e1a4"
format_stamp: "Formatted at 2026-09-26 13:10:35 on dist-test-slave-2dw5"
server_key: "378a21b2ee34f02bbb901fa9eede678b"
server_key_iv: "340c3c47bf88fb423c55137492e679d7"
server_key_version: "kuduclusterkey@0"
I20260926 13:10:35.411649 11681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:10:35.425817 11681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:10:35.428056 11681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:10:35.428192 11681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:10:35.428478 11681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:10:35.428826 11681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:10:35.428862 11681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:10:35.428891 11681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:10:35.428927 11681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:10:35.436224 11681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45171
I20260926 13:10:35.436578 11681 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:10:35.437290 11811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45171 every 8 connection(s)
Sep 26 13:10:35 dist-test-slave-2dw5 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:10:35.444423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 11681
I20260926 13:10:35.444532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:10:35.444757 392 external_mini_cluster.cc:1428] Setting key 1da00b98c41eda0191ba3583c4f44da1
I20260926 13:10:35.447588 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:35.437411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41603 (local address 127.0.98.1:46937)
0926 13:10:35.438977 (+ 1566us) negotiation.cc:107] Waiting for socket to connect
0926 13:10:35.439026 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:10:35.439668 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:10:35.440055 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:10:35.440058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:10:35.440247 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:10:35.440441 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:35.440445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:35.441166 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:35.441167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:35.441952 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:35.441954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:35.442313 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:35.442315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:35.442435 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:35.442953 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:10:35.442961 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:10:35.444200 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:10:35.446107 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:10:35.446115 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:10:35.446118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:10:35.446409 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:10:35.446610 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:10:35.446610 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:10:35.446611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:10:35.446736 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:10:35.447065 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:10:35.447072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:10:35.447228 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 13:10:35.447297 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:10:35.447419 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1516,"thread_start_us":81,"threads_started":1}
I20260926 13:10:35.448035 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:35.437713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46937 (local address 127.0.98.62:41603)
0926 13:10:35.439857 (+ 2144us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:35.439858 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:35.439865 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:10:35.439887 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:35.439887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:35.439899 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:35.439953 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:10:35.440572 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:35.441083 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:35.442032 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:35.442245 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:35.442307 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:35.444326 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:10:35.444328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:10:35.444328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:10:35.444337 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:10:35.446016 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:10:35.446488 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:10:35.446489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:10:35.446489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:10:35.446548 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:10:35.446806 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:10:35.446806 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:10:35.446807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:10:35.446988 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:10:35.447846 (+ 858us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:35.447887 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 13:10:35.447968 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2121,"thread_start_us":76,"threads_started":1}
I20260926 13:10:35.448467 11812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41603
I20260926 13:10:35.448560 11812 heartbeater.cc:464] Registering TS with master...
I20260926 13:10:35.448740 11812 heartbeater.cc:511] Master 127.0.98.62:41603 requested a full tablet report, sending...
I20260926 13:10:35.449254 11600 ts_manager.cc:194] Registered new tserver with Master: 97518660ff8c440f8169f75aba12e1a4 (127.0.98.1:45171)
I20260926 13:10:35.449838 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46937
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:10:35.477872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54611/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41603
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:10:35.579649 11818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:10:35.579860 11818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:10:35.579895 11818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:10:35.581248 11818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:10:35.581302 11818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:10:35.581343 11818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:10:35.583196 11818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54611/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:10:35.583585 11818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:10:35.583847 11818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:10:35.590627 11826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:10:35.590710 11828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:10:35.591023 11818 server_base.cc:1034] running on GCE node
W20260926 13:10:35.590889 11825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:10:35.591461 11818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:10:35.591811 11818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:10:35.592999 11818 hybrid_clock.cc:648] HybridClock initialized: now 1790428235592969 us; error 43 us; skew 500 ppm
Sep 26 13:10:35 dist-test-slave-2dw5 krb5kdc[11124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790428235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 13:10:35 dist-test-slave-2dw5 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:10:35.596846 11818 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:10:35.597328 11818 webserver.cc:460] Webserver started at http://127.0.98.2:46531/ using document root <none> and password file <none>
I20260926 13:10:35.597563 11818 fs_manager.cc:359] Metadata directory not provided
I20260926 13:10:35.597610 11818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:10:35.597751 11818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:10:35 dist-test-slave-2dw5 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:10:35.614113 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "73b6543cff8e4d68974b96aab0c7351c"
format_stamp: "Formatted at 2026-09-26 13:10:35 on dist-test-slave-2dw5"
server_key: "eb24aeb15b5e44690cf0bbd24d880c67"
server_key_iv: "b03d11c1cc5ef33ebb48356bdc82d828"
server_key_version: "kuduclusterkey@0"
I20260926 13:10:35.614701 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "73b6543cff8e4d68974b96aab0c7351c"
format_stamp: "Formatted at 2026-09-26 13:10:35 on dist-test-slave-2dw5"
server_key: "eb24aeb15b5e44690cf0bbd24d880c67"
server_key_iv: "b03d11c1cc5ef33ebb48356bdc82d828"
server_key_version: "kuduclusterkey@0"
I20260926 13:10:35.619295 11818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260926 13:10:35.642956 11835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:10:35.643257 11818 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260926 13:10:35.643368 11818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "73b6543cff8e4d68974b96aab0c7351c"
format_stamp: "Formatted at 2026-09-26 13:10:35 on dist-test-slave-2dw5"
server_key: "eb24aeb15b5e44690cf0bbd24d880c67"
server_key_iv: "b03d11c1cc5ef33ebb48356bdc82d828"
server_key_version: "kuduclusterkey@0"
I20260926 13:10:35.643479 11818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:10:35.674576 11818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:10:35.676970 11818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:10:35.677153 11818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:10:35.677379 11818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:10:35.677762 11818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:10:35.677810 11818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:10:35.677838 11818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:10:35.677861 11818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:10:35.684585 11818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46263
I20260926 13:10:35.685030 11818 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:10:35.685999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 11818
I20260926 13:10:35.686094 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:10:35.686280 392 external_mini_cluster.cc:1428] Setting key c10e849b71746e4326da91f867a2264d
I20260926 13:10:35.695814 11948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46263 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 26 13:10:35 dist-test-slave-2dw5 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:10:35.710902 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:35.697009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55685 (local address 127.0.98.62:41603)
0926 13:10:35.697076 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:35.697079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:35.700198 (+ 3119us) server_negotiation.cc:407] Connection header received
0926 13:10:35.701005 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:35.701008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:35.701022 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:35.701098 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:10:35.701850 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:35.702613 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:35.704138 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:35.704440 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:35.704524 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:35.707380 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:10:35.707382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:10:35.707382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:10:35.707391 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:10:35.709105 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:10:35.709671 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:10:35.709672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:10:35.709672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:10:35.709735 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:10:35.710052 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:10:35.710053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:10:35.710054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:10:35.710213 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:10:35.710264 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:35.710662 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 13:10:35.710768 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 13:10:35.711463 11952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:35.696647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41603 (local address 127.0.98.2:55685)
0926 13:10:35.699991 (+ 3344us) negotiation.cc:107] Waiting for socket to connect
0926 13:10:35.700066 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 13:10:35.700855 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:10:35.701183 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:10:35.701187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:10:35.701379 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:10:35.701727 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:35.701735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:35.702732 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:35.702733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:35.704047 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:35.704051 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:35.704619 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:35.704620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:35.704776 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:35.705442 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:10:35.705453 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:10:35.707240 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:10:35.709223 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:10:35.709235 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:10:35.709238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:10:35.709575 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:10:35.709801 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:10:35.709802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:10:35.709802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:10:35.709958 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:10:35.710299 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:10:35.710307 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:10:35.710503 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 13:10:35.711011 (+ 508us) client_negotiation.cc:240] Negotiation successful
0926 13:10:35.711209 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3268,"thread_start_us":142,"threads_started":1}
I20260926 13:10:35.711968 11949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41603
I20260926 13:10:35.712072 11949 heartbeater.cc:464] Registering TS with master...
I20260926 13:10:35.712288 11949 heartbeater.cc:511] Master 127.0.98.62:41603 requested a full tablet report, sending...
I20260926 13:10:35.712632 11600 ts_manager.cc:194] Registered new tserver with Master: 73b6543cff8e4d68974b96aab0c7351c (127.0.98.2:46263)
I20260926 13:10:35.713160 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:10:35.744545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54611/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41603
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:10:35.872959 11955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:10:35.873366 11955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:10:35.873467 11955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:10:35.875548 11955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:10:35.875738 11955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:10:35.875847 11955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:10:35.878448 11955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54611/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:10:35.878783 11955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:10:35.879221 11955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:10:35.886318 11961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:10:35.886682 11960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:10:35.886902 11963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:10:35.888860 11955 server_base.cc:1034] running on GCE node
I20260926 13:10:35.889173 11955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:10:35.889519 11955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:10:35.890702 11955 hybrid_clock.cc:648] HybridClock initialized: now 1790428235890674 us; error 41 us; skew 500 ppm
Sep 26 13:10:35 dist-test-slave-2dw5 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:10:35.894302 11955 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:10:35.894925 11955 webserver.cc:460] Webserver started at http://127.0.98.3:35361/ using document root <none> and password file <none>
I20260926 13:10:35.895237 11955 fs_manager.cc:359] Metadata directory not provided
I20260926 13:10:35.895304 11955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:10:35.895440 11955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:10:35 dist-test-slave-2dw5 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:10:35.912530 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "bee2ee535b53405e8ae28a0e2f5a48b8"
format_stamp: "Formatted at 2026-09-26 13:10:35 on dist-test-slave-2dw5"
server_key: "60a6e831a0ec7b1712f9071ac154f5a8"
server_key_iv: "4cd6145aee45d1ad009f394b139a140a"
server_key_version: "kuduclusterkey@0"
I20260926 13:10:35.913118 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bee2ee535b53405e8ae28a0e2f5a48b8"
format_stamp: "Formatted at 2026-09-26 13:10:35 on dist-test-slave-2dw5"
server_key: "60a6e831a0ec7b1712f9071ac154f5a8"
server_key_iv: "4cd6145aee45d1ad009f394b139a140a"
server_key_version: "kuduclusterkey@0"
I20260926 13:10:35.919276 11955 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260926 13:10:35.938634 11970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:10:35.938998 11955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:10:35.939085 11955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "bee2ee535b53405e8ae28a0e2f5a48b8"
format_stamp: "Formatted at 2026-09-26 13:10:35 on dist-test-slave-2dw5"
server_key: "60a6e831a0ec7b1712f9071ac154f5a8"
server_key_iv: "4cd6145aee45d1ad009f394b139a140a"
server_key_version: "kuduclusterkey@0"
I20260926 13:10:35.939185 11955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:10:35.971148 11955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:10:35.974591 11955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:10:35.974745 11955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:10:35.975042 11955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:10:35.975401 11955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:10:35.975445 11955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:10:35.975472 11955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:10:35.975487 11955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:10:35.982481 11955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38547
I20260926 13:10:35.985046 12083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38547 every 8 connection(s)
I20260926 13:10:35.985304 11955 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:10:35.990957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 11955
I20260926 13:10:35.991055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:10:35.991277 392 external_mini_cluster.cc:1428] Setting key 4a8cc21b8ac6513d38d32d30eb7edf82
Sep 26 13:10:35 dist-test-slave-2dw5 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:10:35.997479 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:35.986714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39041 (local address 127.0.98.62:41603)
0926 13:10:35.986757 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:35.986759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:35.986994 (+ 235us) server_negotiation.cc:407] Connection header received
0926 13:10:35.987666 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:35.987667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:35.987683 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:35.987770 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:10:35.988464 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:35.989252 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:35.990800 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:35.991123 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:35.991174 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:35.994137 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:10:35.994139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:10:35.994139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:10:35.994147 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:10:35.995825 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:10:35.996357 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:10:35.996358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:10:35.996359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:10:35.996420 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:10:35.996690 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:10:35.996691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:10:35.996691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:10:35.996843 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:10:35.996886 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:35.997286 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 13:10:35.997377 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:10:35.998462 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:35.986452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41603 (local address 127.0.98.3:39041)
0926 13:10:35.986835 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0926 13:10:35.986916 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 13:10:35.987525 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:10:35.987849 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:10:35.987854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:10:35.988089 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:10:35.988350 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:35.988356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:35.989376 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:35.989377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:35.990681 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:35.990685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:35.991399 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:35.991401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:35.991554 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:35.992164 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:10:35.992174 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:10:35.994017 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:10:35.995930 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:10:35.995938 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:10:35.995940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:10:35.996266 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:10:35.996485 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:10:35.996486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:10:35.996486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:10:35.996624 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:10:35.996931 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:10:35.996938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:10:35.997136 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 13:10:35.998028 (+ 892us) client_negotiation.cc:240] Negotiation successful
0926 13:10:35.998214 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":80,"threads_started":1}
I20260926 13:10:35.999143 12084 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41603
I20260926 13:10:35.999254 12084 heartbeater.cc:464] Registering TS with master...
I20260926 13:10:35.999446 12084 heartbeater.cc:511] Master 127.0.98.62:41603 requested a full tablet report, sending...
I20260926 13:10:36.000360 11600 ts_manager.cc:194] Registered new tserver with Master: bee2ee535b53405e8ae28a0e2f5a48b8 (127.0.98.3:38547)
I20260926 13:10:36.000900 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39041
I20260926 13:10:36.002897 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:10:36.013636 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.004531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50648 (local address 127.0.98.62:41603)
0926 13:10:36.004577 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:36.004579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:36.004871 (+ 292us) server_negotiation.cc:407] Connection header received
0926 13:10:36.004970 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:36.004971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:36.004982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:36.005011 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:10:36.006001 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.006816 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.007963 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.008271 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.008311 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.009925 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:10:36.009927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:10:36.009927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:10:36.009935 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:10:36.012251 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:10:36.012719 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:10:36.012720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:10:36.012721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:10:36.012796 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:10:36.013014 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:10:36.013015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:10:36.013015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:10:36.013152 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:10:36.013240 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:36.013448 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 13:10:36.013536 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:10:36.015934 11600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:10:36.018342 11600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:10:36.035378 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.021843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38547 (local address 127.0.0.1:51862)
0926 13:10:36.023388 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0926 13:10:36.023436 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:10:36.023502 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:10:36.024488 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:10:36.024489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:10:36.024490 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:10:36.024521 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.024522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.031983 (+ 7461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.031984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.033169 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.033172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.034956 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.034958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.035062 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.035106 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 13:10:36.035214 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 13:10:36.035280 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1512,"thread_start_us":79,"threads_started":1}
I20260926 13:10:36.039822 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.022046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46263 (local address 127.0.0.1:45432)
0926 13:10:36.023988 (+ 1942us) negotiation.cc:107] Waiting for socket to connect
0926 13:10:36.023997 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:10:36.024041 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:10:36.024352 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:10:36.024355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:10:36.024357 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:10:36.024427 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.024429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.035630 (+ 11201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.035631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.036997 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.037000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.039514 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.039515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.039607 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.039612 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:10:36.039662 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 13:10:36.039712 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1927,"thread_start_us":55,"threads_started":1}
I20260926 13:10:36.040195 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.022328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.0.98.3:38547)
0926 13:10:36.023646 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:36.023651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:36.023660 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:10:36.023698 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:36.023700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:36.023716 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:36.023794 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:10:36.025208 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.031897 (+ 6689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.033865 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.034707 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.034795 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.034918 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:36.040045 (+ 5127us) server_negotiation.cc:299] Negotiation successful
0926 13:10:36.040106 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1280,"thread_start_us":96,"threads_started":1}
I20260926 13:10:36.040963 11999 tablet_service.cc:1467] Processing CreateTablet for tablet 517e8651f43a47359c616f359d81aa6e (DEFAULT_TABLE table=test-table [id=515b8d5efe88458caf6c7b713abd291a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:10:36.041373 11999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517e8651f43a47359c616f359d81aa6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:10:36.046571 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.022507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.0.98.2:46263)
0926 13:10:36.023937 (+ 1430us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:36.023941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:36.024155 (+ 214us) server_negotiation.cc:407] Connection header received
0926 13:10:36.024193 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:36.024195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:36.024209 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:36.024268 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:10:36.024582 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.033364 (+ 8782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.037131 (+ 3767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.039308 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.039388 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.039476 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:36.046397 (+ 6921us) server_negotiation.cc:299] Negotiation successful
0926 13:10:36.046467 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1394,"thread_start_us":59,"threads_started":1}
I20260926 13:10:36.047340 11883 tablet_service.cc:1467] Processing CreateTablet for tablet 517e8651f43a47359c616f359d81aa6e (DEFAULT_TABLE table=test-table [id=515b8d5efe88458caf6c7b713abd291a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:10:36.047763 11883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517e8651f43a47359c616f359d81aa6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:10:36.053424 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.022800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45171 (local address 127.0.0.1:37486)
0926 13:10:36.027571 (+ 4771us) negotiation.cc:107] Waiting for socket to connect
0926 13:10:36.027582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:10:36.027637 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:10:36.027946 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:10:36.027947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:10:36.027949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:10:36.028009 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.028011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.029852 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.029853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.031647 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.031651 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.053119 (+ 21468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.053121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.053212 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.053218 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:10:36.053272 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 13:10:36.053329 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4750,"thread_start_us":77,"threads_started":1}
I20260926 13:10:36.053951 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.023040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37486 (local address 127.0.98.1:45171)
0926 13:10:36.024133 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:36.024137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:36.027702 (+ 3565us) server_negotiation.cc:407] Connection header received
0926 13:10:36.027758 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:36.027761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:36.027780 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:36.027857 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:10:36.028083 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.029732 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.038072 (+ 8340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.052909 (+ 14837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.052996 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.053070 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:36.053815 (+ 745us) server_negotiation.cc:299] Negotiation successful
0926 13:10:36.053878 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063,"thread_start_us":51,"threads_started":1}
I20260926 13:10:36.054486 12103 tablet_bootstrap.cc:492] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8: Bootstrap starting.
I20260926 13:10:36.054685 11746 tablet_service.cc:1467] Processing CreateTablet for tablet 517e8651f43a47359c616f359d81aa6e (DEFAULT_TABLE table=test-table [id=515b8d5efe88458caf6c7b713abd291a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:10:36.055097 11746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517e8651f43a47359c616f359d81aa6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:10:36.055713 12103 tablet_bootstrap.cc:654] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8: Neither blocks nor log segments found. Creating new log.
I20260926 13:10:36.055960 12104 tablet_bootstrap.cc:492] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c: Bootstrap starting.
I20260926 13:10:36.057126 12103 log.cc:824] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:10:36.057611 12104 tablet_bootstrap.cc:654] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c: Neither blocks nor log segments found. Creating new log.
I20260926 13:10:36.057981 12104 log.cc:824] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:10:36.058888 12103 tablet_bootstrap.cc:492] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8: No bootstrap required, opened a new log
I20260926 13:10:36.058964 12103 ts_tablet_manager.cc:1397] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260926 13:10:36.062109 12103 raft_consensus.cc:348] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97518660ff8c440f8169f75aba12e1a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45171 } } peers { permanent_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38547 } } peers { permanent_uuid: "73b6543cff8e4d68974b96aab0c7351c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260926 13:10:36.062315 12103 raft_consensus.cc:374] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:10:36.062501 12103 raft_consensus.cc:729] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bee2ee535b53405e8ae28a0e2f5a48b8, State: Initialized, Role: FOLLOWER
I20260926 13:10:36.062737 12103 consensus_queue.cc:260] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97518660ff8c440f8169f75aba12e1a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45171 } } peers { permanent_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38547 } } peers { permanent_uuid: "73b6543cff8e4d68974b96aab0c7351c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260926 13:10:36.063169 12103 ts_tablet_manager.cc:1428] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260926 13:10:36.063313 12084 heartbeater.cc:503] Master 127.0.98.62:41603 was elected leader, sending a full tablet report...
I20260926 13:10:36.065115 12104 tablet_bootstrap.cc:492] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c: No bootstrap required, opened a new log
I20260926 13:10:36.065208 12104 ts_tablet_manager.cc:1397] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260926 13:10:36.067381 12104 raft_consensus.cc:348] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97518660ff8c440f8169f75aba12e1a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45171 } } peers { permanent_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38547 } } peers { permanent_uuid: "73b6543cff8e4d68974b96aab0c7351c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260926 13:10:36.067621 12104 raft_consensus.cc:374] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:10:36.067672 12104 raft_consensus.cc:729] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b6543cff8e4d68974b96aab0c7351c, State: Initialized, Role: FOLLOWER
I20260926 13:10:36.067835 12104 consensus_queue.cc:260] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97518660ff8c440f8169f75aba12e1a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45171 } } peers { permanent_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38547 } } peers { permanent_uuid: "73b6543cff8e4d68974b96aab0c7351c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260926 13:10:36.068130 12104 ts_tablet_manager.cc:1428] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:10:36.068246 11949 heartbeater.cc:503] Master 127.0.98.62:41603 was elected leader, sending a full tablet report...
I20260926 13:10:36.075063 12107 tablet_bootstrap.cc:492] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4: Bootstrap starting.
I20260926 13:10:36.076057 12107 tablet_bootstrap.cc:654] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4: Neither blocks nor log segments found. Creating new log.
I20260926 13:10:36.076417 12107 log.cc:824] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4: Log is configured to *not* fsync() on all Append() calls
I20260926 13:10:36.077373 12107 tablet_bootstrap.cc:492] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4: No bootstrap required, opened a new log
I20260926 13:10:36.077461 12107 ts_tablet_manager.cc:1397] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:10:36.080241 12107 raft_consensus.cc:348] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97518660ff8c440f8169f75aba12e1a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45171 } } peers { permanent_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38547 } } peers { permanent_uuid: "73b6543cff8e4d68974b96aab0c7351c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260926 13:10:36.080472 12107 raft_consensus.cc:374] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:10:36.080506 12107 raft_consensus.cc:729] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97518660ff8c440f8169f75aba12e1a4, State: Initialized, Role: FOLLOWER
I20260926 13:10:36.080704 12107 consensus_queue.cc:260] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97518660ff8c440f8169f75aba12e1a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45171 } } peers { permanent_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38547 } } peers { permanent_uuid: "73b6543cff8e4d68974b96aab0c7351c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260926 13:10:36.081045 12107 ts_tablet_manager.cc:1428] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 13:10:36.081185 11812 heartbeater.cc:503] Master 127.0.98.62:41603 was elected leader, sending a full tablet report...
Sep 26 13:10:36 dist-test-slave-2dw5 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 13:10:36.188011 11813 tablet.cc:2369] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:10:36.196877 11950 tablet.cc:2369] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:10:36.228225 12108 raft_consensus.cc:482] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:10:36.228381 12108 raft_consensus.cc:504] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97518660ff8c440f8169f75aba12e1a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45171 } } peers { permanent_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38547 } } peers { permanent_uuid: "73b6543cff8e4d68974b96aab0c7351c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260926 13:10:36.228792 12108 leader_election.cc:288] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97518660ff8c440f8169f75aba12e1a4 (127.0.98.1:45171), 73b6543cff8e4d68974b96aab0c7351c (127.0.98.2:46263)
I20260926 13:10:36.232959 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.229520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56909 (local address 127.0.98.2:46263)
0926 13:10:36.229579 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:36.229581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:36.229588 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:10:36.229610 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:36.229611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:36.229620 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:36.229663 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:10:36.229950 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.230617 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.232014 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.232569 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.232616 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.232628 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:36.232845 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 13:10:36.232900 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:10:36.232952 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.229023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45171 (local address 127.0.98.3:56245)
0926 13:10:36.229280 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 13:10:36.229292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:10:36.229339 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:10:36.229659 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:10:36.229660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:10:36.229662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:10:36.229729 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.229731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.230620 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.230621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.231896 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.231900 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.232678 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.232679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.232764 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.232769 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:10:36.232812 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 13:10:36.232856 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":8}
I20260926 13:10:36.233239 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.229439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56245 (local address 127.0.98.1:45171)
0926 13:10:36.229510 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:36.229512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:36.229520 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:10:36.229548 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:36.229549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:36.229560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:36.229610 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:10:36.229797 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.230536 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.232006 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.232586 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.232640 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.232648 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:36.233121 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 13:10:36.233166 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:10:36.233460 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.229151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46263 (local address 127.0.98.3:56909)
0926 13:10:36.229291 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 13:10:36.229299 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:10:36.229339 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:10:36.229816 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:10:36.229816 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:10:36.229818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:10:36.229876 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.229877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.230692 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.230693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.231909 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.231910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.232678 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.232679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.232764 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.232771 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:10:36.233347 (+ 576us) client_negotiation.cc:240] Negotiation successful
0926 13:10:36.233389 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":15,"thread_start_us":85,"threads_started":1}
I20260926 13:10:36.233515 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517e8651f43a47359c616f359d81aa6e" candidate_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97518660ff8c440f8169f75aba12e1a4" is_pre_election: true
I20260926 13:10:36.233701 11765 raft_consensus.cc:2454] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bee2ee535b53405e8ae28a0e2f5a48b8 in term 0.
I20260926 13:10:36.233728 12109 raft_consensus.cc:482] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:10:36.233728 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517e8651f43a47359c616f359d81aa6e" candidate_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b6543cff8e4d68974b96aab0c7351c" is_pre_election: true
I20260926 13:10:36.233820 12109 raft_consensus.cc:504] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97518660ff8c440f8169f75aba12e1a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45171 } } peers { permanent_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38547 } } peers { permanent_uuid: "73b6543cff8e4d68974b96aab0c7351c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260926 13:10:36.233949 11972 leader_election.cc:302] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97518660ff8c440f8169f75aba12e1a4, bee2ee535b53405e8ae28a0e2f5a48b8; no voters:
I20260926 13:10:36.234009 11903 raft_consensus.cc:2454] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bee2ee535b53405e8ae28a0e2f5a48b8 in term 0.
I20260926 13:10:36.234074 12108 raft_consensus.cc:2781] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:10:36.234110 12108 raft_consensus.cc:482] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:10:36.234128 12108 raft_consensus.cc:3037] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:10:36.234122 12109 leader_election.cc:288] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97518660ff8c440f8169f75aba12e1a4 (127.0.98.1:45171), bee2ee535b53405e8ae28a0e2f5a48b8 (127.0.98.3:38547)
I20260926 13:10:36.234982 12108 raft_consensus.cc:504] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97518660ff8c440f8169f75aba12e1a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45171 } } peers { permanent_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38547 } } peers { permanent_uuid: "73b6543cff8e4d68974b96aab0c7351c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260926 13:10:36.235118 12108 leader_election.cc:288] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [CANDIDATE]: Term 1 election: Requested vote from peers 97518660ff8c440f8169f75aba12e1a4 (127.0.98.1:45171), 73b6543cff8e4d68974b96aab0c7351c (127.0.98.2:46263)
I20260926 13:10:36.235368 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517e8651f43a47359c616f359d81aa6e" candidate_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b6543cff8e4d68974b96aab0c7351c"
I20260926 13:10:36.235443 11903 raft_consensus.cc:3037] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:10:36.235601 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517e8651f43a47359c616f359d81aa6e" candidate_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97518660ff8c440f8169f75aba12e1a4"
I20260926 13:10:36.235679 11765 raft_consensus.cc:3037] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:10:36.236326 11903 raft_consensus.cc:2454] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bee2ee535b53405e8ae28a0e2f5a48b8 in term 1.
I20260926 13:10:36.236443 11765 raft_consensus.cc:2454] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bee2ee535b53405e8ae28a0e2f5a48b8 in term 1.
W20260926 13:10:36.236491 12085 tablet.cc:2369] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:10:36.236810 11971 leader_election.cc:302] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b6543cff8e4d68974b96aab0c7351c, bee2ee535b53405e8ae28a0e2f5a48b8; no voters:
I20260926 13:10:36.237170 12108 raft_consensus.cc:2781] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:10:36.237322 12108 raft_consensus.cc:686] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 1 LEADER]: Becoming Leader. State: Replica: bee2ee535b53405e8ae28a0e2f5a48b8, State: Running, Role: LEADER
I20260926 13:10:36.237416 12108 consensus_queue.cc:237] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97518660ff8c440f8169f75aba12e1a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45171 } } peers { permanent_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38547 } } peers { permanent_uuid: "73b6543cff8e4d68974b96aab0c7351c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260926 13:10:36.238234 11600 catalog_manager.cc:5495] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bee2ee535b53405e8ae28a0e2f5a48b8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97518660ff8c440f8169f75aba12e1a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73b6543cff8e4d68974b96aab0c7351c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } health_report { overall_health: UNKNOWN } } }
I20260926 13:10:36.247839 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.236901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42215 (local address 127.0.98.3:38547)
0926 13:10:36.236975 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:36.236977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:36.244193 (+ 7216us) server_negotiation.cc:407] Connection header received
0926 13:10:36.244257 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:36.244258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:36.244268 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:36.244332 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:10:36.244567 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.245317 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.246869 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.247415 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.247461 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.247474 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:36.247705 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 13:10:36.247759 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:10:36.247833 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.235474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45279 (local address 127.0.98.1:45171)
0926 13:10:36.238184 (+ 2710us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:36.238186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:36.244331 (+ 6145us) server_negotiation.cc:407] Connection header received
0926 13:10:36.244427 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:36.244429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:36.244441 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:36.244467 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:10:36.244617 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.245303 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.246870 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.247447 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.247482 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.247490 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:36.247704 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 13:10:36.247754 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2687}
I20260926 13:10:36.248087 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.235290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45171 (local address 127.0.98.2:45279)
0926 13:10:36.244277 (+ 8987us) negotiation.cc:107] Waiting for socket to connect
0926 13:10:36.244285 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:10:36.244358 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:10:36.244517 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:10:36.244517 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:10:36.244519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:10:36.244554 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.244555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.245407 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.245408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.246731 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.246733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.247536 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.247537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.247626 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.247629 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:10:36.247940 (+ 311us) client_negotiation.cc:240] Negotiation successful
0926 13:10:36.248005 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8977,"mutex_wait_us":33,"thread_start_us":69,"threads_started":1}
I20260926 13:10:36.248339 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.234267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38547 (local address 127.0.98.2:42215)
0926 13:10:36.244101 (+ 9834us) negotiation.cc:107] Waiting for socket to connect
0926 13:10:36.244115 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:10:36.244196 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:10:36.244402 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:10:36.244403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:10:36.244405 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:10:36.244492 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.244494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.245414 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.245415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.246729 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:10:36.246733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.247536 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.247536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:10:36.247625 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.247629 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:10:36.248227 (+ 598us) client_negotiation.cc:240] Negotiation successful
0926 13:10:36.248272 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9814,"mutex_wait_us":94,"thread_start_us":63,"threads_started":1}
I20260926 13:10:36.248358 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517e8651f43a47359c616f359d81aa6e" candidate_uuid: "73b6543cff8e4d68974b96aab0c7351c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97518660ff8c440f8169f75aba12e1a4" is_pre_election: true
I20260926 13:10:36.248489 11765 raft_consensus.cc:2379] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 73b6543cff8e4d68974b96aab0c7351c in current term 1: Already voted for candidate bee2ee535b53405e8ae28a0e2f5a48b8 in this term.
I20260926 13:10:36.248652 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517e8651f43a47359c616f359d81aa6e" candidate_uuid: "73b6543cff8e4d68974b96aab0c7351c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bee2ee535b53405e8ae28a0e2f5a48b8" is_pre_election: true
I20260926 13:10:36.248916 11836 leader_election.cc:302] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73b6543cff8e4d68974b96aab0c7351c; no voters: 97518660ff8c440f8169f75aba12e1a4, bee2ee535b53405e8ae28a0e2f5a48b8
I20260926 13:10:36.249040 12109 raft_consensus.cc:2726] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 13:10:36.254902 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.252332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.0.98.3:38547)
0926 13:10:36.252371 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 13:10:36.252373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:10:36.252381 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:10:36.252453 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:10:36.252453 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:10:36.252470 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:10:36.252520 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:10:36.252641 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.253099 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.254153 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:10:36.254356 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:10:36.254418 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:10:36.254586 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:10:36.254691 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:10:36.254770 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:10:36.254786 (+ 16us) server_negotiation.cc:299] Negotiation successful
0926 13:10:36.254805 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:10:36.259868 11765 raft_consensus.cc:1261] T 517e8651f43a47359c616f359d81aa6e P 97518660ff8c440f8169f75aba12e1a4 [term 1 FOLLOWER]: Refusing update from remote peer bee2ee535b53405e8ae28a0e2f5a48b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:10:36.259897 11903 raft_consensus.cc:1261] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [term 1 FOLLOWER]: Refusing update from remote peer bee2ee535b53405e8ae28a0e2f5a48b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:10:36.260210 12115 consensus_queue.cc:1035] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b6543cff8e4d68974b96aab0c7351c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:10:36.260399 12108 consensus_queue.cc:1035] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97518660ff8c440f8169f75aba12e1a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:10:36.264158 12118 mvcc.cc:205] Tried to move back new op lower bound from 7333594055717302272 to 7333594055628730368. Current Snapshot: MvccSnapshot[applied={T|T < 7333594055717302272}]
I20260926 13:10:36.267333 12119 mvcc.cc:205] Tried to move back new op lower bound from 7333594055717302272 to 7333594055628730368. Current Snapshot: MvccSnapshot[applied={T|T < 7333594055717302272}]
I20260926 13:10:36.267776 12120 mvcc.cc:205] Tried to move back new op lower bound from 7333594055717302272 to 7333594055628730368. Current Snapshot: MvccSnapshot[applied={T|T < 7333594055717302272}]
I20260926 13:10:36.270588 11599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:10:36.270685 11599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:10:36.271449 11599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a2cf64b712a4ba6be0ac483fb4a9154: Sending DeleteTablet for 3 replicas of tablet 517e8651f43a47359c616f359d81aa6e
I20260926 13:10:36.271946 11999 tablet_service.cc:1514] Processing DeleteTablet for tablet 517e8651f43a47359c616f359d81aa6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:10:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51862
I20260926 13:10:36.272190 11883 tablet_service.cc:1514] Processing DeleteTablet for tablet 517e8651f43a47359c616f359d81aa6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:10:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45432
I20260926 13:10:36.272356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 11681
I20260926 13:10:36.272460 11746 tablet_service.cc:1514] Processing DeleteTablet for tablet 517e8651f43a47359c616f359d81aa6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:10:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37486
I20260926 13:10:36.276130 12129 tablet_replica.cc:330] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c: stopping tablet replica
I20260926 13:10:36.276307 12128 tablet_replica.cc:330] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8: stopping tablet replica
I20260926 13:10:36.276458 12128 raft_consensus.cc:2229] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:10:36.276613 12129 raft_consensus.cc:2229] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:10:36.276803 12128 raft_consensus.cc:2258] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:10:36.276955 12129 raft_consensus.cc:2258] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:10:36.277711 12128 ts_tablet_manager.cc:1905] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:10:36.277999 12129 ts_tablet_manager.cc:1905] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:10:36.281205 12129 ts_tablet_manager.cc:1918] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:10:36.281406 12128 ts_tablet_manager.cc:1918] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:10:36.281519 12128 log.cc:1196] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/517e8651f43a47359c616f359d81aa6e
I20260926 13:10:36.281838 12128 ts_tablet_manager.cc:1939] T 517e8651f43a47359c616f359d81aa6e P bee2ee535b53405e8ae28a0e2f5a48b8: Deleting consensus metadata
I20260926 13:10:36.281965 12129 log.cc:1196] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/517e8651f43a47359c616f359d81aa6e
I20260926 13:10:36.282572 12129 ts_tablet_manager.cc:1939] T 517e8651f43a47359c616f359d81aa6e P 73b6543cff8e4d68974b96aab0c7351c: Deleting consensus metadata
I20260926 13:10:36.283123 11584 catalog_manager.cc:4841] TS bee2ee535b53405e8ae28a0e2f5a48b8 (127.0.98.3:38547): tablet 517e8651f43a47359c616f359d81aa6e (table test-table [id=515b8d5efe88458caf6c7b713abd291a]) successfully deleted
I20260926 13:10:36.287550 11584 catalog_manager.cc:4841] TS 73b6543cff8e4d68974b96aab0c7351c (127.0.98.2:46263): tablet 517e8651f43a47359c616f359d81aa6e (table test-table [id=515b8d5efe88458caf6c7b713abd291a]) successfully deleted
W20260926 13:10:36.291236 11585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45171 (error 108)
I20260926 13:10:36.292258 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:10:36.292121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45171 (local address 127.0.0.1:37496)
0926 13:10:36.292173 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 13:10:36.292219 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 13:10:36.292511 11585 catalog_manager.cc:4584] TS 97518660ff8c440f8169f75aba12e1a4 (127.0.98.1:45171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 517e8651f43a47359c616f359d81aa6e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45171: connect: Connection refused (error 111)
I20260926 13:10:36.292786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 11818
I20260926 13:10:36.300253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 11955
I20260926 13:10:36.305121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 11568
I20260926 13:10:36.311380 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 13:10:36.646759 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 13:10:36.646963 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 13:10:36.981869 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 13:10:36.982 UTC [11154] LOG: received smart shutdown request
2026-09-26 13:10:36.985 UTC [11154] LOG: background worker "logical replication launcher" (PID 11163) exited with exit code 1
2026-09-26 13:10:36.985 UTC [11158] LOG: shutting down
2026-09-26 13:10:37.182 UTC [11154] LOG: database system is shut down
2026-09-26T13:10:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46754 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:10:37 dist-test-slave-2dw5 krb5kdc[12139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:10:37 dist-test-slave-2dw5 krb5kdc[12139](info): set up 2 sockets
Sep 26 13:10:37 dist-test-slave-2dw5 krb5kdc[12139](info): commencing operation
krb5kdc: starting...
W20260926 13:10:38.235033 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:10:38 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:10:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:10:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 13:10:38.410725 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasky34iiX/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/postgres -l logfile start
2026-09-26 13:10:41.146 UTC [12169] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 13:10:41.146 UTC [12169] LOG: listening on IPv4 address "127.0.98.20", port 43747
2026-09-26 13:10:41.150 UTC [12169] LOG: listening on Unix socket "/tmp/.s.PGSQL.43747"
2026-09-26 13:10:41.174 UTC [12172] LOG: database system was shut down at 2026-09-26 13:10:39 UTC
2026-09-26 13:10:41.178 UTC [12169] LOG: database system is ready to accept connections
I20260926 13:10:42.046525 392 mini_postgres.cc:96] Postgres bound to 43747
2026-09-26 13:10:42.052 UTC [12180] FATAL: database "slave" does not exist
127.0.98.20:43747 - accepting connections
I20260926 13:10:42.052950 392 mini_ranger.cc:161] Starting Ranger...
I20260926 13:10:42.067140 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 13:10:42.612891 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 13:10:42.612993 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-admin
2026-09-26 13:10:42,915 [I] DB FLAVOR :POSTGRES
2026-09-26 13:10:42,916 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:10:42,916 [I] Checking connection..
2026-09-26 13:10:42,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 13:10:43,196 [I] Checking connection passed.
2026-09-26 13:10:43,197 [I] --------- Verifying version history table ---------
2026-09-26 13:10:43,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:10:43,454 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:10:43,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:10:43,695 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:10:43,695 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 13:10:43,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 13:10:43.930 UTC [12244] WARNING: there is no transaction in progress
2026-09-26 13:10:43,936 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 13:10:43,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:10:44,203 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 13:10:44,203 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:10:44,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 13:10:44,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 13:10:44,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2dw5.c.gcp-upstream.internal','N') ;"
2026-09-26 13:10:44,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:10:45,176 [I] Table x_portal_user does not exist in database ranger
2026-09-26 13:10:45,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:10:45,417 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 13:10:45,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:10:45,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:10:45,913 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 13:10:45,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 13:10:46.979 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:46.998 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.018 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.037 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.055 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.075 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.137 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.145 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.161 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.173 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.184 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.199 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.211 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.218 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.227 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:47.235 UTC [12352] WARNING: there is no transaction in progress
2026-09-26 13:10:48,003 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 13:10:48,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 13:10:48,237 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 13:10:48,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:10:48,476 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 13:10:48,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:10:48,731 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 13:10:48,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:10:49,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:10:49,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2dw5.c.gcp-upstream.internal';"
2026-09-26 13:10:49,510 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 13:10:49,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 13:10:49,773 [I] DB_PATCHES have already been applied
I20260926 13:10:49.779469 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-admin:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasky34iiX/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 13:10:49.779534 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 13:10:49.782560 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:53845
Sep 26, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 13:10:50.804877 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.752s user 0.000s sys 0.008s
Sep 26, 2026 1:11:11 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:11:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 1:11:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:11:12 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:11:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 13:11:13.597584 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 13:11:13 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428273, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 13:11:13.684387 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 13:11:13.693920 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 13:11:14.766503 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 13:11:14.766589 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-kms
I20260926 13:11:14.766605 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:43747
2026-09-26 13:11:14,928 [I] DB FLAVOR :POSTGRES
2026-09-26 13:11:14,928 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:11:14,928 [I] Checking connection
2026-09-26 13:11:14,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 13:11:15,182 [I] connection success
2026-09-26 13:11:15,182 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 13:11:15,182 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 13:11:15,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 13:11:15,450 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 13:11:15,450 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:11:15,451 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 13:11:15,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43747/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 13:11:15,742 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 13:11:16.210301 392 mini_ranger_kms.cc:310] Created kms service
I20260926 13:11:16.388967 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 13:11:16.436226 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 13:11:16 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428273, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:11:16.866289 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 13:11:16.866398 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-kms:/tmp/dist-test-tasky34iiX/build/release/bin/postgresql.jar:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasky34iiX/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 13:11:16.866415 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 13:11:16.870167 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12533
I20260926 13:11:16.870253 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:44467
Sep 26, 2026 1:11:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:11:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:11:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:11:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:11:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:11:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 13:11:18.954595 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.270s user 0.001s sys 0.141s
I20260926 13:11:18.954685 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 13:11:18.954734 392 mini_ranger_kms.cc:401] 127.0.98.20:44467/kms/v1/keys
Sep 26 13:11:19 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790428279, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 13:11:19 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428279, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 13:11:20 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428279, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 13:11:22 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:11:22.490687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39399
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39399
--ranger_config_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44467/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:22.574000 12583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:22.574240 12583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:22.574270 12583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:22.575474 12583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:11:22.575520 12583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:22.575538 12583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:22.575553 12583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:11:22.575568 12583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:11:22.577185 12583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:44467/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39399
--ranger_config_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:22.577426 12583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:22.577626 12583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:22.584244 12588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:22.584435 12589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:22.584301 12583 server_base.cc:1034] running on GCE node
W20260926 13:11:22.584276 12591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:22.584885 12583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:22.585245 12583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:22.586413 12583 hybrid_clock.cc:648] HybridClock initialized: now 1790428282586391 us; error 36 us; skew 500 ppm
Sep 26 13:11:22 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:22.598196 12583 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:11:22.598786 12583 webserver.cc:460] Webserver started at http://127.0.98.62:34341/ using document root <none> and password file <none>
I20260926 13:11:22.599038 12583 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:22.599083 12583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:22.599213 12583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:11:22 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:11:22.650681 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "e5f8c65c64be4a86a32fad08691a45a0"
format_stamp: "Formatted at 2026-09-26 13:11:22 on dist-test-slave-2dw5"
server_key: "b78037c83829984b94695684751a51f8"
server_key_iv: "0f088f82f3f55aedee0bb458e9e92da8"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:22.651144 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "e5f8c65c64be4a86a32fad08691a45a0"
format_stamp: "Formatted at 2026-09-26 13:11:22 on dist-test-slave-2dw5"
server_key: "b78037c83829984b94695684751a51f8"
server_key_iv: "0f088f82f3f55aedee0bb458e9e92da8"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:22.652479 12583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:22.666805 12598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:22.667090 12583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:22.667217 12583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "e5f8c65c64be4a86a32fad08691a45a0"
format_stamp: "Formatted at 2026-09-26 13:11:22 on dist-test-slave-2dw5"
server_key: "b78037c83829984b94695684751a51f8"
server_key_iv: "0f088f82f3f55aedee0bb458e9e92da8"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:22.667316 12583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:22.697917 12583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:22.699890 12583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:22.700050 12583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:22.704547 12583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39399
I20260926 13:11:22.704623 12650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39399 every 8 connection(s)
I20260926 13:11:22.704972 12583 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:11:22.705624 12651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:22.707337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 12583
I20260926 13:11:22.707435 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:11:22.707651 392 external_mini_cluster.cc:1428] Setting key 9daa1de21203b261be437cae5f307bd2
I20260926 13:11:22.708163 12651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0: Bootstrap starting.
I20260926 13:11:22.708830 12651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:22.709126 12651 log.cc:824] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:22.709909 12651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0: No bootstrap required, opened a new log
I20260926 13:11:22.712107 12651 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8c65c64be4a86a32fad08691a45a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39399 } }
I20260926 13:11:22.712299 12651 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:22.712330 12651 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f8c65c64be4a86a32fad08691a45a0, State: Initialized, Role: FOLLOWER
I20260926 13:11:22.712488 12651 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8c65c64be4a86a32fad08691a45a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39399 } }
I20260926 13:11:22.712571 12651 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:11:22.712601 12651 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:11:22.712659 12651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:22.713196 12651 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8c65c64be4a86a32fad08691a45a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39399 } }
I20260926 13:11:22.713305 12651 leader_election.cc:302] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f8c65c64be4a86a32fad08691a45a0; no voters:
I20260926 13:11:22.713502 12651 leader_election.cc:288] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:11:22.713550 12656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:11:22.713768 12656 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 1 LEADER]: Becoming Leader. State: Replica: e5f8c65c64be4a86a32fad08691a45a0, State: Running, Role: LEADER
I20260926 13:11:22.713878 12651 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:11:22.713883 12656 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8c65c64be4a86a32fad08691a45a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39399 } }
Sep 26 13:11:22 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:22.715795 12657 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5f8c65c64be4a86a32fad08691a45a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8c65c64be4a86a32fad08691a45a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39399 } } }
I20260926 13:11:22.715869 12657 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:22.716049 12658 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f8c65c64be4a86a32fad08691a45a0. Latest consensus state: current_term: 1 leader_uuid: "e5f8c65c64be4a86a32fad08691a45a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8c65c64be4a86a32fad08691a45a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39399 } } }
I20260926 13:11:22.716112 12658 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:22.716578 12651 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 13:11:22.716702 12670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:11:22.719244 12670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:11:22.724205 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:22.708226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.0.98.62:39399)
0926 13:11:22.708417 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:22.708427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:22.708458 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:11:22.709006 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:22.709016 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:22.709092 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:22.709252 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:22.709932 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:22.711354 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:22.712500 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:22.713211 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:22.713323 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:22.716359 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:22.716364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:22.716364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:22.716380 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:22.720782 (+ 4402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:22.721467 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:22.721471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:22.721471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:22.721645 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:22.722913 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:22.722915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:22.722915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:22.723337 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:22.723490 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:22.723621 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 13:11:22.723778 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
I20260926 13:11:22.727407 12670 catalog_manager.cc:1347] Generated new cluster ID: 9e88a202900b4f43b02f98fba26e0d0f
I20260926 13:11:22.727463 12670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:11:22.737021 12670 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:11:22.737850 12670 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:11:22.741616 12670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0: Generated new TSK 0
I20260926 13:11:22.741892 12670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:11:23.531327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39399
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:23.651568 12697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:23.651865 12697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:23.651899 12697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:23.653752 12697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:23.653824 12697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:23.653864 12697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:11:23.656250 12697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:44467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:23.656642 12697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:23.656924 12697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:23.663101 12702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:23.663111 12705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:23.663111 12703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:23.663790 12697 server_base.cc:1034] running on GCE node
I20260926 13:11:23.663998 12697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:23.664253 12697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:23.665416 12697 hybrid_clock.cc:648] HybridClock initialized: now 1790428283665398 us; error 31 us; skew 500 ppm
Sep 26 13:11:23 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:23.668637 12697 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:11:23.669225 12697 webserver.cc:460] Webserver started at http://127.0.98.1:44807/ using document root <none> and password file <none>
I20260926 13:11:23.669487 12697 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:23.669551 12697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:23.669695 12697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:11:23 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:11:23.684315 12697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "d5d5dbb7eeef44ca871473c6dc1c8483"
format_stamp: "Formatted at 2026-09-26 13:11:23 on dist-test-slave-2dw5"
server_key: "22b3b1c5bcc5c567c3a8b02b3f69229c"
server_key_iv: "74821a3991cfced74673e79aad4a96c2"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:23.684866 12697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d5d5dbb7eeef44ca871473c6dc1c8483"
format_stamp: "Formatted at 2026-09-26 13:11:23 on dist-test-slave-2dw5"
server_key: "22b3b1c5bcc5c567c3a8b02b3f69229c"
server_key_iv: "74821a3991cfced74673e79aad4a96c2"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:23.687268 12697 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:11:23.704186 12712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:23.707207 12697 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260926 13:11:23.707343 12697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "d5d5dbb7eeef44ca871473c6dc1c8483"
format_stamp: "Formatted at 2026-09-26 13:11:23 on dist-test-slave-2dw5"
server_key: "22b3b1c5bcc5c567c3a8b02b3f69229c"
server_key_iv: "74821a3991cfced74673e79aad4a96c2"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:23.707461 12697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:23.718703 12697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:23.720575 12697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:23.720710 12697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:23.720940 12697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:23.721274 12697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:23.721312 12697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:23.721341 12697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:23.721378 12697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:23.728907 12697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43751
I20260926 13:11:23.729451 12697 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:11:23.732146 12826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43751 every 8 connection(s)
I20260926 13:11:23.733644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 12697
I20260926 13:11:23.733741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:11:23.733958 392 external_mini_cluster.cc:1428] Setting key 08999bef96efef4de9829a01154308b6
Sep 26 13:11:23 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:23.749320 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:23.732932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39485 (local address 127.0.98.62:39399)
0926 13:11:23.734530 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:23.734532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:23.736358 (+ 1826us) server_negotiation.cc:407] Connection header received
0926 13:11:23.737119 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:23.737121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:23.737138 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:23.737187 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:23.738075 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:23.738872 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:23.740445 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:23.740717 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:23.740765 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:23.744368 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:23.744370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:23.744370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:23.744406 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:23.746209 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:23.746924 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:23.746925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:23.746926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:23.747001 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:23.748365 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:23.748367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:23.748367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:23.748546 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:23.748613 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:23.749079 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 13:11:23.749191 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1557,"thread_start_us":110,"threads_started":1}
I20260926 13:11:23.750160 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:23.732977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39399 (local address 127.0.98.1:39485)
0926 13:11:23.734617 (+ 1640us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:23.734670 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:23.736979 (+ 2309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:23.737408 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:23.737414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:23.737674 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:23.737957 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:23.737964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:23.738994 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:23.738995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:23.740330 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:23.740334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:23.740890 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:23.740891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:23.741035 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:23.742512 (+ 1477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:23.742523 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:23.744226 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:23.746331 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:23.746340 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:23.746343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:23.746660 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:23.747078 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:23.747079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:23.747079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:23.747242 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:23.748680 (+ 1438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:23.748689 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:23.748901 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 13:11:23.749719 (+ 818us) client_negotiation.cc:240] Negotiation successful
0926 13:11:23.749913 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1589,"thread_start_us":101,"threads_started":1}
I20260926 13:11:23.751914 12827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39399
I20260926 13:11:23.752017 12827 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:23.752208 12827 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
I20260926 13:11:23.752856 12615 ts_manager.cc:194] Registered new tserver with Master: d5d5dbb7eeef44ca871473c6dc1c8483 (127.0.98.1:43751)
I20260926 13:11:23.753736 12615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39485
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:11:23.782752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39399
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:23.901201 12834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:23.901607 12834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:23.901703 12834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:23.903772 12834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:23.903910 12834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:23.904002 12834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:11:23.905951 12834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:44467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:23.906216 12834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:23.906549 12834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:23.913662 12844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:23.913663 12842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:23.914177 12841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:23.915619 12834 server_base.cc:1034] running on GCE node
I20260926 13:11:23.915897 12834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:23.916193 12834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:23.917383 12834 hybrid_clock.cc:648] HybridClock initialized: now 1790428283917358 us; error 36 us; skew 500 ppm
Sep 26 13:11:23 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:23.921180 12834 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:11:23.921768 12834 webserver.cc:460] Webserver started at http://127.0.98.2:34107/ using document root <none> and password file <none>
I20260926 13:11:23.922032 12834 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:23.922083 12834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:23.922212 12834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:11:23 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:11:23.937161 12834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "534f5795c890437f860ddf04256c397a"
format_stamp: "Formatted at 2026-09-26 13:11:23 on dist-test-slave-2dw5"
server_key: "a74d3c41a4c6643fd13d1d3a368f7f2b"
server_key_iv: "54608b395db337cce650c842657f9eb0"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:23.937647 12834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "534f5795c890437f860ddf04256c397a"
format_stamp: "Formatted at 2026-09-26 13:11:23 on dist-test-slave-2dw5"
server_key: "a74d3c41a4c6643fd13d1d3a368f7f2b"
server_key_iv: "54608b395db337cce650c842657f9eb0"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:23.939728 12834 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
Sep 26 13:11:23 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790428283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:23.955655 12851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:23.955873 12834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 13:11:23.955958 12834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "534f5795c890437f860ddf04256c397a"
format_stamp: "Formatted at 2026-09-26 13:11:23 on dist-test-slave-2dw5"
server_key: "a74d3c41a4c6643fd13d1d3a368f7f2b"
server_key_iv: "54608b395db337cce650c842657f9eb0"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:23.956050 12834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:23.966712 12834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:23.969372 12834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:23.969631 12834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:23.969890 12834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:23.970341 12834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:23.970525 12834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:23.970609 12834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:23.970666 12834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:23.977090 12834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34893
I20260926 13:11:23.977690 12834 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:11:23.978453 12964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34893 every 8 connection(s)
I20260926 13:11:23.979379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 12834
I20260926 13:11:23.979472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:11:23.979667 392 external_mini_cluster.cc:1428] Setting key 8d67166b8eec4e15fb1737101ca55501
Sep 26 13:11:23 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:23.994509 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:23.979929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35823 (local address 127.0.98.62:39399)
0926 13:11:23.979990 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:23.979992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:23.980161 (+ 169us) server_negotiation.cc:407] Connection header received
0926 13:11:23.980962 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:23.980963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:23.980975 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:23.981003 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:23.981813 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:23.982604 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:23.984199 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:23.984497 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:23.984544 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:23.987181 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:23.987183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:23.987183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:23.987192 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:23.988943 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:23.989434 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:23.989436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:23.989436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:23.989510 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:23.991956 (+ 2446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:23.991958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:23.991958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:23.992123 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:23.992175 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:23.994224 (+ 2049us) server_negotiation.cc:299] Negotiation successful
0926 13:11:23.994398 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:11:23.995393 12967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:23.979626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39399 (local address 127.0.98.2:35823)
0926 13:11:23.980026 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:23.980073 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:23.980839 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:23.981181 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:23.981185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:23.981441 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:23.981709 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:23.981714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:23.982718 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:23.982719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:23.984099 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:23.984103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:23.984648 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:23.984649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:23.984785 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:23.985402 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:23.985413 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:23.987069 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:23.989023 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:23.989031 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:23.989033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:23.989334 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:23.989611 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:23.989612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:23.989612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:23.989762 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:23.992229 (+ 2467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:23.992238 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:23.992441 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 13:11:23.994906 (+ 2465us) client_negotiation.cc:240] Negotiation successful
0926 13:11:23.995125 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":99,"threads_started":1}
I20260926 13:11:23.996094 12965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39399
I20260926 13:11:23.996194 12965 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:23.996407 12965 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
I20260926 13:11:23.996858 12615 ts_manager.cc:194] Registered new tserver with Master: 534f5795c890437f860ddf04256c397a (127.0.98.2:34893)
I20260926 13:11:23.997414 12615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35823
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:11:24.025610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39399
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:24.148290 12970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:24.148670 12970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:24.148751 12970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:24.150519 12970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:24.150748 12970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:24.150835 12970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:11:24.153208 12970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:44467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:24.153704 12970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:24.154012 12970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:24.160648 12976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:24.161026 12979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:24.161180 12977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:24.162184 12970 server_base.cc:1034] running on GCE node
I20260926 13:11:24.162565 12970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:24.162833 12970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:24.164043 12970 hybrid_clock.cc:648] HybridClock initialized: now 1790428284164023 us; error 31 us; skew 500 ppm
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:24.167282 12970 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:11:24.167802 12970 webserver.cc:460] Webserver started at http://127.0.98.3:41467/ using document root <none> and password file <none>
I20260926 13:11:24.168047 12970 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:24.168092 12970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:24.168210 12970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:11:24.182896 12970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "7db3fb1e8a604b21b8256b7f8fcc8512"
format_stamp: "Formatted at 2026-09-26 13:11:24 on dist-test-slave-2dw5"
server_key: "b25c9f6c6a6c6323a873e6c8319ec1b6"
server_key_iv: "fb0383f45b0e9996891414ba36f77bf1"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:24.183470 12970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7db3fb1e8a604b21b8256b7f8fcc8512"
format_stamp: "Formatted at 2026-09-26 13:11:24 on dist-test-slave-2dw5"
server_key: "b25c9f6c6a6c6323a873e6c8319ec1b6"
server_key_iv: "fb0383f45b0e9996891414ba36f77bf1"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:24.191357 12970 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.000s
I20260926 13:11:24.208093 12986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:24.208343 12970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:11:24.208446 12970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "7db3fb1e8a604b21b8256b7f8fcc8512"
format_stamp: "Formatted at 2026-09-26 13:11:24 on dist-test-slave-2dw5"
server_key: "b25c9f6c6a6c6323a873e6c8319ec1b6"
server_key_iv: "fb0383f45b0e9996891414ba36f77bf1"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:24.208545 12970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:24.222047 12970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:24.229566 12970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:24.229792 12970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:24.230055 12970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:24.231689 12970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:24.231798 12970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:24.231885 12970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:24.231945 12970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:24.238256 12970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36971
I20260926 13:11:24.238772 12970 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:11:24.241629 13099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36971 every 8 connection(s)
I20260926 13:11:24.242360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 12970
I20260926 13:11:24.242482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:11:24.242681 392 external_mini_cluster.cc:1428] Setting key 9876b546404649098259cce21bb4eb9c
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:24.260035 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:24.242427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51981 (local address 127.0.98.62:39399)
0926 13:11:24.242864 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:24.242866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:24.246275 (+ 3409us) server_negotiation.cc:407] Connection header received
0926 13:11:24.250207 (+ 3932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:24.250209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:24.250225 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:24.250259 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:24.251138 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:24.251846 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:24.253384 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:24.253702 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:24.253745 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:24.256767 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:24.256769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:24.256769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:24.256778 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:24.258308 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:24.258883 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:24.258884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:24.258884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:24.258953 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:24.259237 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:24.259238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:24.259238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:24.259390 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:24.259467 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:24.259835 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 13:11:24.259936 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381}
I20260926 13:11:24.260247 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:24.242410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39399 (local address 127.0.98.3:51981)
0926 13:11:24.246104 (+ 3694us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:24.246171 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:24.247074 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:24.250501 (+ 3427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:24.250507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:24.250766 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:24.251036 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:24.251042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:24.251976 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:24.251978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:24.253257 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:24.253261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:24.253963 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:24.253964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:24.254117 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:24.254819 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:24.254896 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:24.256657 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:24.258407 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:24.258416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:24.258419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:24.258759 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:24.259010 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:24.259011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:24.259011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:24.259169 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:24.259454 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:24.259459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:24.259671 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 13:11:24.259781 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 13:11:24.259945 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3630,"thread_start_us":106,"threads_started":1}
I20260926 13:11:24.260917 13100 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39399
I20260926 13:11:24.261029 13100 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:24.261230 13100 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
I20260926 13:11:24.261617 12615 ts_manager.cc:194] Registered new tserver with Master: 7db3fb1e8a604b21b8256b7f8fcc8512 (127.0.98.3:36971)
I20260926 13:11:24.262068 12615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51981
I20260926 13:11:24.265681 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:11:24.266444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 12697
I20260926 13:11:24.271903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 12834
I20260926 13:11:24.278494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 12970
I20260926 13:11:24.286983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 12583
I20260926 13:11:24.292101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39399
--webserver_interface=127.0.98.62
--webserver_port=34341
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39399
--ranger_config_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44467/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44467/kms with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:24.374215 13103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:24.374441 13103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:24.374472 13103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:24.375686 13103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:11:24.375735 13103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:24.375754 13103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:24.375769 13103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:11:24.375787 13103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:11:24.377451 13103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44467/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39399
--ranger_config_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:24.377694 13103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:24.377905 13103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:24.382577 13109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:24.382707 13111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:24.382573 13108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:24.382872 13103 server_base.cc:1034] running on GCE node
I20260926 13:11:24.383090 13103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:24.383356 13103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:24.384507 13103 hybrid_clock.cc:648] HybridClock initialized: now 1790428284384485 us; error 36 us; skew 500 ppm
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:24.387209 13103 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:11:24.387656 13103 webserver.cc:460] Webserver started at http://127.0.98.62:34341/ using document root <none> and password file <none>
I20260926 13:11:24.387861 13103 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:24.387913 13103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:11:24.459455 13103 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:11:24.460220 13119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:24.460412 13103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:11:24.460494 13103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "e5f8c65c64be4a86a32fad08691a45a0"
format_stamp: "Formatted at 2026-09-26 13:11:22 on dist-test-slave-2dw5"
server_key: "b78037c83829984b94695684751a51f8"
server_key_iv: "0f088f82f3f55aedee0bb458e9e92da8"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:24.460875 13103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:24.490010 13103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:24.492146 13103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:24.492270 13103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:24.496413 13103 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39399
I20260926 13:11:24.496469 13171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39399 every 8 connection(s)
I20260926 13:11:24.496815 13103 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:11:24.497849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 13103
I20260926 13:11:24.498097 13172 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:11:24.498185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43751
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39399
--webserver_port=44807
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44467/kms with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:11:24.498873 13172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0: Bootstrap starting.
I20260926 13:11:24.506886 13172 log.cc:824] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:24.508680 13172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:11:24.508988 13172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0: Bootstrap complete.
I20260926 13:11:24.511303 13172 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8c65c64be4a86a32fad08691a45a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39399 } }
I20260926 13:11:24.511574 13172 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f8c65c64be4a86a32fad08691a45a0, State: Initialized, Role: FOLLOWER
I20260926 13:11:24.511729 13172 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8c65c64be4a86a32fad08691a45a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39399 } }
I20260926 13:11:24.511801 13172 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:11:24.511859 13172 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:11:24.511905 13172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:11:24.512972 13172 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8c65c64be4a86a32fad08691a45a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39399 } }
I20260926 13:11:24.513079 13172 leader_election.cc:302] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f8c65c64be4a86a32fad08691a45a0; no voters:
I20260926 13:11:24.513249 13172 leader_election.cc:288] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:11:24.513319 13176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:11:24.513501 13172 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:11:24.513526 13176 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [term 2 LEADER]: Becoming Leader. State: Replica: e5f8c65c64be4a86a32fad08691a45a0, State: Running, Role: LEADER
I20260926 13:11:24.513721 13176 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8c65c64be4a86a32fad08691a45a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39399 } }
I20260926 13:11:24.514006 13177 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e5f8c65c64be4a86a32fad08691a45a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8c65c64be4a86a32fad08691a45a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39399 } } }
I20260926 13:11:24.514024 13178 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f8c65c64be4a86a32fad08691a45a0. Latest consensus state: current_term: 2 leader_uuid: "e5f8c65c64be4a86a32fad08691a45a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f8c65c64be4a86a32fad08691a45a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39399 } } }
I20260926 13:11:24.514174 13177 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:24.514182 13178 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:24.514384 13185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:11:24.514987 13185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:11:24.515162 13185 catalog_manager.cc:1259] Loaded cluster ID: 9e88a202900b4f43b02f98fba26e0d0f
I20260926 13:11:24.515235 13185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:11:24.515300 13172 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 13:11:24.518666 13185 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:11:24.519182 13185 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0: Loaded TSK: 0
I20260926 13:11:24.519492 13185 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:11:24.583861 13174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:24.584106 13174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:24.584144 13174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:24.585438 13174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:24.585505 13174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:24.585546 13174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:11:24.587556 13174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44807
--tserver_master_addrs=127.0.98.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:24.587966 13174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:24.588264 13174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:24.594651 13208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:24.594789 13207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:24.594655 13210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:24.595413 13174 server_base.cc:1034] running on GCE node
I20260926 13:11:24.595645 13174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:24.595952 13174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:24.597144 13174 hybrid_clock.cc:648] HybridClock initialized: now 1790428284597127 us; error 28 us; skew 500 ppm
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:24.600618 13174 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:11:24.601200 13174 webserver.cc:460] Webserver started at http://127.0.98.1:44807/ using document root <none> and password file <none>
I20260926 13:11:24.601468 13174 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:24.601554 13174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:11:24.621706 13174 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:24.622654 13220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:24.622954 13174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:24.623131 13174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "d5d5dbb7eeef44ca871473c6dc1c8483"
format_stamp: "Formatted at 2026-09-26 13:11:23 on dist-test-slave-2dw5"
server_key: "22b3b1c5bcc5c567c3a8b02b3f69229c"
server_key_iv: "74821a3991cfced74673e79aad4a96c2"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:24.623706 13174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:24.640326 13174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:24.643005 13174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:24.643172 13174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:24.643486 13174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:24.643884 13174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:24.643967 13174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:24.644018 13174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:24.644049 13174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:24.650207 13174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43751
I20260926 13:11:24.650817 13174 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:11:24.654983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 13174
I20260926 13:11:24.655277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34893
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39399
--webserver_port=34107
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44467/kms with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:11:24.659577 13333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43751 every 8 connection(s)
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:24.689989 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:24.660359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39399 (local address 127.0.98.1:56633)
0926 13:11:24.667072 (+ 6713us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:24.667147 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:24.667831 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:24.673235 (+ 5404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:24.673240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:24.673518 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:24.673829 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:24.673837 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:24.678487 (+ 4650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:24.678488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:24.681069 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:24.681075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:24.683063 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:24.683065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:24.683217 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:24.683917 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:24.683928 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:24.685671 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:24.687969 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:24.687979 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:24.687981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:24.688294 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:24.688603 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:24.688604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:24.688604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:24.688737 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:24.689218 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:24.689225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:24.689480 (+ 255us) client_negotiation.cc:769] Sending connection context
0926 13:11:24.689580 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 13:11:24.689754 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6650,"thread_start_us":109,"threads_started":1}
I20260926 13:11:24.691321 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:24.660449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56633 (local address 127.0.98.62:39399)
0926 13:11:24.662979 (+ 2530us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:24.662989 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:24.668328 (+ 5339us) server_negotiation.cc:407] Connection header received
0926 13:11:24.668533 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:24.668542 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:24.668609 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:24.672107 (+ 3498us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:24.673987 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:24.678284 (+ 4297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:24.681209 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:24.681892 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:24.682009 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:24.685834 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:24.685840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:24.685841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:24.685856 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:24.687862 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:24.688395 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:24.688400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:24.688400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:24.688528 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:24.688805 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:24.688806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:24.688807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:24.689109 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:24.689179 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:24.690971 (+ 1792us) server_negotiation.cc:299] Negotiation successful
0926 13:11:24.691118 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2465,"thread_start_us":99,"threads_started":1}
I20260926 13:11:24.692487 13334 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39399
I20260926 13:11:24.692579 13334 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:24.692752 13334 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
W20260926 13:11:24.693660 13334 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39399 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 13:11:24.693727 13334 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:24.693786 13334 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
I20260926 13:11:24.694209 13334 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:24.694281 13334 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
W20260926 13:11:24.694615 13334 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:11:24.773445 13336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:24.773680 13336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:24.773705 13336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:24.775182 13336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:24.775236 13336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:24.775266 13336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:11:24.776859 13336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34107
--tserver_master_addrs=127.0.98.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:24.777266 13336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:24.777572 13336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:24.784200 13346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:24.784200 13344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:24.784219 13343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:24.784422 13336 server_base.cc:1034] running on GCE node
I20260926 13:11:24.784726 13336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:24.785037 13336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:24.786227 13336 hybrid_clock.cc:648] HybridClock initialized: now 1790428284786207 us; error 33 us; skew 500 ppm
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:24.789659 13336 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:11:24.790503 13336 webserver.cc:460] Webserver started at http://127.0.98.2:34107/ using document root <none> and password file <none>
I20260926 13:11:24.790817 13336 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:24.790937 13336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:11:24.814976 13336 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260926 13:11:24.817660 13354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:24.817943 13336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:11:24.818051 13336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "534f5795c890437f860ddf04256c397a"
format_stamp: "Formatted at 2026-09-26 13:11:23 on dist-test-slave-2dw5"
server_key: "a74d3c41a4c6643fd13d1d3a368f7f2b"
server_key_iv: "54608b395db337cce650c842657f9eb0"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:24.818431 13336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:24.835822 13336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:24.838459 13336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:24.838629 13336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:24.838919 13336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:24.839311 13336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:24.839363 13336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:24.839395 13336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:24.839412 13336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:24.847534 13336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34893
I20260926 13:11:24.847613 13467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34893 every 8 connection(s)
I20260926 13:11:24.848047 13336 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:24.855618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 13336
I20260926 13:11:24.855937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36971
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39399
--webserver_port=41467
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44467/kms with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:11:24.857709 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:24.848951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55709 (local address 127.0.98.62:39399)
0926 13:11:24.849005 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:24.849007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:24.849248 (+ 241us) server_negotiation.cc:407] Connection header received
0926 13:11:24.849803 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:24.849804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:24.849817 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:24.849875 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:24.850454 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:24.851062 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:24.852046 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:24.852329 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:24.852383 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:24.854513 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:24.854515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:24.854516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:24.854524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:24.856097 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:24.856609 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:24.856610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:24.856611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:24.856668 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:24.856916 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:24.856917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:24.856917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:24.857056 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:24.857096 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:24.857519 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 13:11:24.857609 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:11:24.858481 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:24.848940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39399 (local address 127.0.98.2:55709)
0926 13:11:24.849124 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:24.849161 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:24.849706 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:24.849950 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:24.849956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:24.850166 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:24.850363 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:24.850367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:24.851143 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:24.851144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:24.851963 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:24.851965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:24.852427 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:24.852428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:24.852551 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:24.853063 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:24.853072 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:24.854392 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:24.856193 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:24.856204 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:24.856207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:24.856526 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:24.856729 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:24.856730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:24.856730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:24.856856 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:24.857143 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:24.857151 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:24.857365 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 13:11:24.858056 (+ 691us) client_negotiation.cc:240] Negotiation successful
0926 13:11:24.858243 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260926 13:11:24.859216 13468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39399
I20260926 13:11:24.859310 13468 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:24.859503 13468 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
W20260926 13:11:24.860225 13468 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39399 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 13:11:24.860302 13468 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:24.860363 13468 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
I20260926 13:11:24.860756 13468 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:24.860828 13468 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
W20260926 13:11:24.861145 13468 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:11:24.949750 13471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:24.950065 13471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:24.950106 13471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:24.952042 13471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:24.952111 13471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:24.952154 13471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:11:24.954487 13471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:44467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41467
--tserver_master_addrs=127.0.98.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:24.955116 13471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:24.955459 13471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:24.962118 13477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:24.962582 13476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:24.962805 13471 server_base.cc:1034] running on GCE node
W20260926 13:11:24.962940 13479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:24.963292 13471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:24.963560 13471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:24.964702 13471 hybrid_clock.cc:648] HybridClock initialized: now 1790428284964670 us; error 46 us; skew 500 ppm
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:24.967895 13471 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:11:24.968470 13471 webserver.cc:460] Webserver started at http://127.0.98.3:41467/ using document root <none> and password file <none>
I20260926 13:11:24.968731 13471 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:24.968801 13471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:11:24 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:11:24.987506 13471 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:24.991019 13487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:24.991271 13471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:11:24.991365 13471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "7db3fb1e8a604b21b8256b7f8fcc8512"
format_stamp: "Formatted at 2026-09-26 13:11:24 on dist-test-slave-2dw5"
server_key: "b25c9f6c6a6c6323a873e6c8319ec1b6"
server_key_iv: "fb0383f45b0e9996891414ba36f77bf1"
server_key_version: "kuduclusterkey@0"
I20260926 13:11:24.991838 13471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:25.016402 13471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:25.022328 13471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:25.022581 13471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:25.022905 13471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:25.023357 13471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:25.023458 13471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:25.023528 13471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:25.023581 13471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:25.030233 13471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36971
I20260926 13:11:25.030681 13601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36971 every 8 connection(s)
I20260926 13:11:25.030798 13471 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:11:25.033007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 13471
Sep 26 13:11:25 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:25.045804 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:25.033437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.0.98.62:39399)
0926 13:11:25.034186 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:25.034188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:25.034396 (+ 208us) server_negotiation.cc:407] Connection header received
0926 13:11:25.034420 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:25.034421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:25.034434 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:25.034496 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:25.035301 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:25.036073 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:25.036938 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:25.037175 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:25.037242 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:25.038640 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:25.038642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:25.038642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:25.038651 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:25.040592 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:25.044117 (+ 3525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:25.044119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:25.044119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:25.044195 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:25.044738 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:25.044739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:25.044739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:25.044901 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:25.045101 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:25.045589 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 13:11:25.045682 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":62,"threads_started":1}
I20260926 13:11:25.046882 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:25.031704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38797 (local address 127.0.98.62:39399)
0926 13:11:25.031755 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:25.031758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:25.032104 (+ 346us) server_negotiation.cc:407] Connection header received
0926 13:11:25.032721 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:25.032722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:25.032735 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:25.032812 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:25.034125 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:25.034912 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:25.036562 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:25.036848 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:25.036897 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:25.043421 (+ 6524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:25.043423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:25.043424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:25.043435 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:25.045097 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:25.045661 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:25.045662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:25.045662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:25.045720 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:25.046039 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:25.046040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:25.046040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:25.046167 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:25.046213 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:25.046670 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 13:11:25.046759 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:11:25.047482 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:25.031744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39399 (local address 127.0.98.3:38797)
0926 13:11:25.031947 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:25.031994 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:25.032595 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:25.033005 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:25.033010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:25.033691 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:25.034019 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:25.034026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:25.035035 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:25.035036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:25.036455 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:25.036458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:25.037016 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:25.037017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:25.037168 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:25.037756 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:25.037766 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:25.043279 (+ 5513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:25.045194 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:25.045207 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:25.045210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:25.045573 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:25.045793 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:25.045794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:25.045794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:25.045953 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:25.046266 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:25.046274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:25.046496 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 13:11:25.047020 (+ 524us) client_negotiation.cc:240] Negotiation successful
0926 13:11:25.047212 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":106,"threads_started":1}
I20260926 13:11:25.048650 13602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39399
I20260926 13:11:25.048760 13602 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:25.048969 13602 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
W20260926 13:11:25.049695 13602 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39399 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 13:11:25.049775 13602 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:25.049844 13602 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
I20260926 13:11:25.050235 13602 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:25.050313 13602 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
W20260926 13:11:25.050665 13602 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 13:11:25 dist-test-slave-2dw5 krb5kdc[12139](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790428285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:25.694813 13334 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:25.695003 13334 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
I20260926 13:11:25.695504 13136 ts_manager.cc:194] Registered new tserver with Master: d5d5dbb7eeef44ca871473c6dc1c8483 (127.0.98.1:43751)
I20260926 13:11:25.696487 13136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56633
I20260926 13:11:25.861342 13468 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:25.861493 13468 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
I20260926 13:11:25.861925 13136 ts_manager.cc:194] Registered new tserver with Master: 534f5795c890437f860ddf04256c397a (127.0.98.2:34893)
I20260926 13:11:25.862485 13136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55709
Sep 26 13:11:25 dist-test-slave-2dw5 krb5kdc[12139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:11:26.050877 13602 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:26.051033 13602 heartbeater.cc:511] Master 127.0.98.62:39399 requested a full tablet report, sending...
I20260926 13:11:26.051352 13136 ts_manager.cc:194] Registered new tserver with Master: 7db3fb1e8a604b21b8256b7f8fcc8512 (127.0.98.3:36971)
I20260926 13:11:26.051920 13136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38797
I20260926 13:11:26.060043 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:11:26.069368 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.061107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.98.62:39399)
0926 13:11:26.061346 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:26.061348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:26.061499 (+ 151us) server_negotiation.cc:407] Connection header received
0926 13:11:26.061524 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:26.061524 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:26.061534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:26.061585 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:26.062282 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.063130 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.064284 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.064618 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.064658 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:26.066157 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:26.066158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:26.066158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:26.066166 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:26.067904 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:26.068278 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:26.068279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:26.068279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:26.068339 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:26.068513 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:26.068514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:26.068514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:26.068640 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:26.068740 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:26.069177 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 13:11:26.069261 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":61,"threads_started":1}
I20260926 13:11:26.071735 13136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:11:26.073976 13136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:11:26.091858 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.085595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43751 (local address 127.0.0.1:58266)
0926 13:11:26.086584 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:26.086590 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:26.086621 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:26.087012 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:26.087016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:26.087018 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:26.087103 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:26.087105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.088717 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.088719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:26.090078 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:26.090383 (+ 305us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.091509 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.091511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:26.091608 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:26.091625 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 13:11:26.091694 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:11:26.091754 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":49,"threads_started":1}
I20260926 13:11:26.092465 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.085710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.0.98.1:43751)
0926 13:11:26.086743 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:26.086749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:26.086759 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:11:26.086793 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:26.086795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:26.086811 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:26.086908 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:26.087183 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.088630 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.090458 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.091304 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.091387 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:26.091471 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:26.092330 (+ 859us) server_negotiation.cc:299] Negotiation successful
0926 13:11:26.092382 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":46,"threads_started":1}
I20260926 13:11:26.093423 13260 tablet_service.cc:1467] Processing CreateTablet for tablet 336535ab32bb45afa4540245f0ceb26c (DEFAULT_TABLE table=test-table [id=aa032e89a9db4358bdea2a0a2b77c965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:11:26.093803 13260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 336535ab32bb45afa4540245f0ceb26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:26.099354 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.085212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36971 (local address 127.0.0.1:57620)
0926 13:11:26.086265 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:26.086315 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:26.086378 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:26.095184 (+ 8806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:26.095186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:26.095190 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:26.095252 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:26.095254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.096751 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.096753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:26.097997 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:26.098002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.099092 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.099093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:26.099163 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:26.099170 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:11:26.099200 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 13:11:26.099251 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":103,"threads_started":1}
I20260926 13:11:26.100054 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.085771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.0.98.3:36971)
0926 13:11:26.094951 (+ 9180us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:26.094956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:26.094971 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:11:26.095010 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:26.095012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:26.095045 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:26.095370 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:26.095452 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.096646 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.098137 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.098972 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.099779 (+ 807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:26.099885 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:26.099913 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 13:11:26.099959 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9134,"thread_start_us":47,"threads_started":1}
I20260926 13:11:26.101114 13536 tablet_service.cc:1467] Processing CreateTablet for tablet 336535ab32bb45afa4540245f0ceb26c (DEFAULT_TABLE table=test-table [id=aa032e89a9db4358bdea2a0a2b77c965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:11:26.101481 13536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 336535ab32bb45afa4540245f0ceb26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:26.106000 13637 tablet_bootstrap.cc:492] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512: Bootstrap starting.
I20260926 13:11:26.106825 13637 tablet_bootstrap.cc:654] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:26.107262 13637 log.cc:824] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:26.108120 13637 tablet_bootstrap.cc:492] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512: No bootstrap required, opened a new log
I20260926 13:11:26.108186 13637 ts_tablet_manager.cc:1397] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:11:26.110831 13637 raft_consensus.cc:348] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "534f5795c890437f860ddf04256c397a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34893 } } peers { permanent_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43751 } }
I20260926 13:11:26.111052 13637 raft_consensus.cc:374] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:26.111086 13637 raft_consensus.cc:729] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db3fb1e8a604b21b8256b7f8fcc8512, State: Initialized, Role: FOLLOWER
I20260926 13:11:26.111909 13637 consensus_queue.cc:260] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "534f5795c890437f860ddf04256c397a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34893 } } peers { permanent_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43751 } }
I20260926 13:11:26.112527 13637 ts_tablet_manager.cc:1428] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260926 13:11:26.112692 13602 heartbeater.cc:503] Master 127.0.98.62:39399 was elected leader, sending a full tablet report...
I20260926 13:11:26.113737 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.085456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34893 (local address 127.0.0.1:52532)
0926 13:11:26.086472 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:26.086479 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:26.086509 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:26.100485 (+ 13976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:26.100486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:26.100490 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:26.100539 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:26.100541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.110970 (+ 10429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.110972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:26.112240 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:26.112245 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.113448 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.113449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:26.113556 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:26.113564 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 13:11:26.113598 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 13:11:26.113652 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":59,"threads_started":1}
I20260926 13:11:26.114193 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.085881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.0.98.2:34893)
0926 13:11:26.100308 (+ 14427us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:26.100313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:26.100326 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:11:26.100362 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:26.100364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:26.100382 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:26.100607 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:26.100681 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.101751 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.112369 (+ 10618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.113334 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.113886 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:26.114001 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:26.114041 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 13:11:26.114116 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14385,"thread_start_us":55,"threads_started":1}
I20260926 13:11:26.114432 13636 tablet_bootstrap.cc:492] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483: Bootstrap starting.
I20260926 13:11:26.114969 13402 tablet_service.cc:1467] Processing CreateTablet for tablet 336535ab32bb45afa4540245f0ceb26c (DEFAULT_TABLE table=test-table [id=aa032e89a9db4358bdea2a0a2b77c965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:11:26.115293 13636 tablet_bootstrap.cc:654] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:26.115329 13402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 336535ab32bb45afa4540245f0ceb26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:26.115597 13636 log.cc:824] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:26.120376 13641 tablet_bootstrap.cc:492] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a: Bootstrap starting.
I20260926 13:11:26.121338 13636 tablet_bootstrap.cc:492] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483: No bootstrap required, opened a new log
I20260926 13:11:26.121523 13636 ts_tablet_manager.cc:1397] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260926 13:11:26.124588 13636 raft_consensus.cc:348] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "534f5795c890437f860ddf04256c397a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34893 } } peers { permanent_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43751 } }
I20260926 13:11:26.124877 13636 raft_consensus.cc:374] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:26.124954 13636 raft_consensus.cc:729] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5d5dbb7eeef44ca871473c6dc1c8483, State: Initialized, Role: FOLLOWER
I20260926 13:11:26.125154 13636 consensus_queue.cc:260] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "534f5795c890437f860ddf04256c397a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34893 } } peers { permanent_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43751 } }
I20260926 13:11:26.125517 13636 ts_tablet_manager.cc:1428] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260926 13:11:26.125684 13334 heartbeater.cc:503] Master 127.0.98.62:39399 was elected leader, sending a full tablet report...
I20260926 13:11:26.126777 13641 tablet_bootstrap.cc:654] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:26.127249 13641 log.cc:824] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:26.132236 13641 tablet_bootstrap.cc:492] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a: No bootstrap required, opened a new log
I20260926 13:11:26.132326 13641 ts_tablet_manager.cc:1397] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260926 13:11:26.134970 13641 raft_consensus.cc:348] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "534f5795c890437f860ddf04256c397a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34893 } } peers { permanent_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43751 } }
I20260926 13:11:26.135167 13641 raft_consensus.cc:374] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:26.135196 13641 raft_consensus.cc:729] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 534f5795c890437f860ddf04256c397a, State: Initialized, Role: FOLLOWER
I20260926 13:11:26.135345 13641 consensus_queue.cc:260] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "534f5795c890437f860ddf04256c397a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34893 } } peers { permanent_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43751 } }
I20260926 13:11:26.135638 13641 ts_tablet_manager.cc:1428] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 13:11:26.135751 13468 heartbeater.cc:503] Master 127.0.98.62:39399 was elected leader, sending a full tablet report...
W20260926 13:11:26.161445 13335 tablet.cc:2369] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:11:26.215770 13639 raft_consensus.cc:482] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:11:26.215931 13639 raft_consensus.cc:504] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "534f5795c890437f860ddf04256c397a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34893 } } peers { permanent_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43751 } }
I20260926 13:11:26.216318 13639 leader_election.cc:288] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 534f5795c890437f860ddf04256c397a (127.0.98.2:34893), d5d5dbb7eeef44ca871473c6dc1c8483 (127.0.98.1:43751)
I20260926 13:11:26.220355 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.216629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53887 (local address 127.0.98.2:34893)
0926 13:11:26.216672 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:26.216674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:26.216925 (+ 251us) server_negotiation.cc:407] Connection header received
0926 13:11:26.217022 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:26.217023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:26.217034 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:26.217074 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:26.217292 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.217976 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.219404 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.219966 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.220011 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:26.220021 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:26.220237 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 13:11:26.220281 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:11:26.220793 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.217163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44749 (local address 127.0.98.1:43751)
0926 13:11:26.217208 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:26.217210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:26.217332 (+ 122us) server_negotiation.cc:407] Connection header received
0926 13:11:26.217432 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:26.217434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:26.217444 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:26.217471 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:26.217727 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.218413 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.219791 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.220367 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.220415 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:26.220425 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:26.220674 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 13:11:26.220723 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:11:26.220805 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.216615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34893 (local address 127.0.98.3:53887)
0926 13:11:26.216848 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:26.216859 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:26.216954 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:26.217139 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:26.217140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:26.217142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:26.217229 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:26.217231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.218077 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.218078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:26.219303 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:26.219306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.220087 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.220088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:26.220158 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:26.220162 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:11:26.220663 (+ 501us) client_negotiation.cc:240] Negotiation successful
0926 13:11:26.220707 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":173,"threads_started":1}
I20260926 13:11:26.221040 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.216925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43751 (local address 127.0.98.3:44749)
0926 13:11:26.217265 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:26.217275 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:26.217358 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:26.217532 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:26.217532 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:26.217534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:26.217643 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:26.217645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.218533 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.218534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:26.219689 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:26.219692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.220494 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.220495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:26.220578 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:26.220581 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:11:26.220918 (+ 337us) client_negotiation.cc:240] Negotiation successful
0926 13:11:26.220961 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":72,"threads_started":1}
I20260926 13:11:26.221369 13269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "336535ab32bb45afa4540245f0ceb26c" candidate_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483" is_pre_election: true
I20260926 13:11:26.221372 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "336535ab32bb45afa4540245f0ceb26c" candidate_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534f5795c890437f860ddf04256c397a" is_pre_election: true
I20260926 13:11:26.221535 13269 raft_consensus.cc:2454] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7db3fb1e8a604b21b8256b7f8fcc8512 in term 0.
I20260926 13:11:26.221535 13422 raft_consensus.cc:2454] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7db3fb1e8a604b21b8256b7f8fcc8512 in term 0.
I20260926 13:11:26.221781 13490 leader_election.cc:302] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db3fb1e8a604b21b8256b7f8fcc8512, d5d5dbb7eeef44ca871473c6dc1c8483; no voters:
I20260926 13:11:26.221971 13639 raft_consensus.cc:2781] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:11:26.222021 13639 raft_consensus.cc:482] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:11:26.222043 13639 raft_consensus.cc:3037] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:26.222915 13639 raft_consensus.cc:504] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "534f5795c890437f860ddf04256c397a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34893 } } peers { permanent_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43751 } }
I20260926 13:11:26.223045 13639 leader_election.cc:288] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [CANDIDATE]: Term 1 election: Requested vote from peers 534f5795c890437f860ddf04256c397a (127.0.98.2:34893), d5d5dbb7eeef44ca871473c6dc1c8483 (127.0.98.1:43751)
I20260926 13:11:26.223233 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "336535ab32bb45afa4540245f0ceb26c" candidate_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "534f5795c890437f860ddf04256c397a"
I20260926 13:11:26.223325 13422 raft_consensus.cc:3037] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:26.224257 13422 raft_consensus.cc:2454] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7db3fb1e8a604b21b8256b7f8fcc8512 in term 1.
I20260926 13:11:26.224403 13490 leader_election.cc:302] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 534f5795c890437f860ddf04256c397a, 7db3fb1e8a604b21b8256b7f8fcc8512; no voters:
I20260926 13:11:26.224471 13269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "336535ab32bb45afa4540245f0ceb26c" candidate_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483"
I20260926 13:11:26.224517 13639 raft_consensus.cc:2781] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:11:26.224558 13269 raft_consensus.cc:3037] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:26.224665 13639 raft_consensus.cc:686] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 1 LEADER]: Becoming Leader. State: Replica: 7db3fb1e8a604b21b8256b7f8fcc8512, State: Running, Role: LEADER
I20260926 13:11:26.224757 13639 consensus_queue.cc:237] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "534f5795c890437f860ddf04256c397a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34893 } } peers { permanent_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43751 } }
I20260926 13:11:26.225435 13269 raft_consensus.cc:2454] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7db3fb1e8a604b21b8256b7f8fcc8512 in term 1.
I20260926 13:11:26.225553 13135 catalog_manager.cc:5495] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7db3fb1e8a604b21b8256b7f8fcc8512 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7db3fb1e8a604b21b8256b7f8fcc8512" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "534f5795c890437f860ddf04256c397a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43751 } health_report { overall_health: UNKNOWN } } }
I20260926 13:11:26.239897 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.236824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.0.98.3:36971)
0926 13:11:26.236916 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:26.236917 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:26.236925 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:11:26.236963 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:26.236964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:26.236986 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:26.237049 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:11:26.237229 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.237798 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.238951 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:26.239236 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:26.239307 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:26.239472 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:11:26.239618 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:11:26.239706 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:26.239791 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 13:11:26.239816 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 13:11:26.241734 13422 raft_consensus.cc:1261] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a [term 1 FOLLOWER]: Refusing update from remote peer 7db3fb1e8a604b21b8256b7f8fcc8512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:11:26.241734 13269 raft_consensus.cc:1261] T 336535ab32bb45afa4540245f0ceb26c P d5d5dbb7eeef44ca871473c6dc1c8483 [term 1 FOLLOWER]: Refusing update from remote peer 7db3fb1e8a604b21b8256b7f8fcc8512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:11:26.242080 13639 consensus_queue.cc:1035] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "534f5795c890437f860ddf04256c397a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:11:26.242206 13647 consensus_queue.cc:1035] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5d5dbb7eeef44ca871473c6dc1c8483" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:11:26.246255 13652 mvcc.cc:205] Tried to move back new op lower bound from 7333594260443435008 to 7333594260376948736. Current Snapshot: MvccSnapshot[applied={T|T < 7333594260443435008}]
I20260926 13:11:26.246547 13650 mvcc.cc:205] Tried to move back new op lower bound from 7333594260443435008 to 7333594260376948736. Current Snapshot: MvccSnapshot[applied={T|T < 7333594260443435008}]
I20260926 13:11:26.249289 13135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:11:26.249392 13135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:11:26.250195 13135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5f8c65c64be4a86a32fad08691a45a0: Sending DeleteTablet for 3 replicas of tablet 336535ab32bb45afa4540245f0ceb26c
I20260926 13:11:26.250702 13536 tablet_service.cc:1514] Processing DeleteTablet for tablet 336535ab32bb45afa4540245f0ceb26c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57620
I20260926 13:11:26.250953 13402 tablet_service.cc:1514] Processing DeleteTablet for tablet 336535ab32bb45afa4540245f0ceb26c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52532
I20260926 13:11:26.251178 13260 tablet_service.cc:1514] Processing DeleteTablet for tablet 336535ab32bb45afa4540245f0ceb26c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58266
I20260926 13:11:26.252739 13651 mvcc.cc:205] Tried to move back new op lower bound from 7333594260443435008 to 7333594260376948736. Current Snapshot: MvccSnapshot[applied={T|T < 7333594260443435008}]
I20260926 13:11:26.252969 13659 tablet_replica.cc:330] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512: stopping tablet replica
I20260926 13:11:26.253073 13659 raft_consensus.cc:2229] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:11:26.253165 13659 raft_consensus.cc:2258] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:11:26.253293 13660 tablet_replica.cc:330] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a: stopping tablet replica
I20260926 13:11:26.253383 13660 raft_consensus.cc:2229] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:11:26.253437 13660 raft_consensus.cc:2258] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:11:26.253539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 13174
I20260926 13:11:26.259648 13660 ts_tablet_manager.cc:1905] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:11:26.263168 13660 ts_tablet_manager.cc:1918] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:11:26.263267 13660 log.cc:1196] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/336535ab32bb45afa4540245f0ceb26c
I20260926 13:11:26.263619 13660 ts_tablet_manager.cc:1939] T 336535ab32bb45afa4540245f0ceb26c P 534f5795c890437f860ddf04256c397a: Deleting consensus metadata
I20260926 13:11:26.264130 13122 catalog_manager.cc:4841] TS 534f5795c890437f860ddf04256c397a (127.0.98.2:34893): tablet 336535ab32bb45afa4540245f0ceb26c (table test-table [id=aa032e89a9db4358bdea2a0a2b77c965]) successfully deleted
W20260926 13:11:26.264454 13122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43751 (error 108)
I20260926 13:11:26.265045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 13336
I20260926 13:11:26.266713 13659 ts_tablet_manager.cc:1905] T 336535ab32bb45afa4540245f0ceb26c P 7db3fb1e8a604b21b8256b7f8fcc8512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:11:26.270622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 13471
I20260926 13:11:26.272006 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.271591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43751 (local address 127.0.0.1:58272)
0926 13:11:26.271911 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:26.271964 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
W20260926 13:11:26.272184 13122 catalog_manager.cc:4584] TS d5d5dbb7eeef44ca871473c6dc1c8483 (127.0.98.1:43751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 336535ab32bb45afa4540245f0ceb26c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43751: connect: Connection refused (error 111)
I20260926 13:11:26.277355 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:26.277222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36971 (local address 127.0.0.1:57646)
0926 13:11:26.277280 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:26.277307 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 13:11:26.277431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 13103
I20260926 13:11:26.282424 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 13:11:26.617558 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 13:11:26.617713 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 13:11:26.963788 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 13:11:26.964 UTC [12169] LOG: received smart shutdown request
2026-09-26 13:11:26.968 UTC [12169] LOG: background worker "logical replication launcher" (PID 12178) exited with exit code 1
2026-09-26 13:11:26.968 UTC [12173] LOG: shutting down
2026-09-26 13:11:27.165 UTC [12169] LOG: database system is shut down
2026-09-26T13:11:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49985 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:11:27 dist-test-slave-2dw5 krb5kdc[13673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:11:27 dist-test-slave-2dw5 krb5kdc[13673](info): set up 2 sockets
Sep 26 13:11:27 dist-test-slave-2dw5 krb5kdc[13673](info): commencing operation
krb5kdc: starting...
W20260926 13:11:28.222985 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:11:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:11:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:11:28.343796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36785
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36785,127.0.98.61:41181,127.0.98.60:37803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:28.425043 13686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:28.425275 13686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:28.425309 13686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:28.426540 13686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:11:28.426587 13686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:28.426605 13686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:28.426622 13686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:11:28.426637 13686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:11:28.428274 13686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36785,127.0.98.61:41181,127.0.98.60:37803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:28.428535 13686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:28.428751 13686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:28.433753 13692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:28.433881 13686 server_base.cc:1034] running on GCE node
W20260926 13:11:28.433753 13691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:28.433753 13694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:28.434341 13686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:28.434621 13686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:28.435767 13686 hybrid_clock.cc:648] HybridClock initialized: now 1790428288435747 us; error 31 us; skew 500 ppm
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:28.444128 13686 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:11:28.444569 13686 webserver.cc:460] Webserver started at http://127.0.98.62:36601/ using document root <none> and password file <none>
I20260926 13:11:28.444787 13686 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:28.444834 13686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:28.444958 13686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:11:28.446025 13686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "65fe39829a6f4561a6b7f7e3c6749ced"
format_stamp: "Formatted at 2026-09-26 13:11:28 on dist-test-slave-2dw5"
server_key: "941869aded58dff743c94e01b1ada076"
server_key_iv: "0577460d81d3381966faf7f08bed5d68"
server_key_version: "encryptionkey@0"
I20260926 13:11:28.446359 13686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "65fe39829a6f4561a6b7f7e3c6749ced"
format_stamp: "Formatted at 2026-09-26 13:11:28 on dist-test-slave-2dw5"
server_key: "941869aded58dff743c94e01b1ada076"
server_key_iv: "0577460d81d3381966faf7f08bed5d68"
server_key_version: "encryptionkey@0"
I20260926 13:11:28.447664 13686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:11:28.448338 13701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:28.448493 13686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:11:28.448573 13686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "65fe39829a6f4561a6b7f7e3c6749ced"
format_stamp: "Formatted at 2026-09-26 13:11:28 on dist-test-slave-2dw5"
server_key: "941869aded58dff743c94e01b1ada076"
server_key_iv: "0577460d81d3381966faf7f08bed5d68"
server_key_version: "encryptionkey@0"
I20260926 13:11:28.448635 13686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:28.457307 13686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:28.459128 13686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:28.459263 13686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:28.462942 13686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36785
I20260926 13:11:28.462975 13753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36785 every 8 connection(s)
I20260926 13:11:28.463310 13686 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:11:28.464121 13754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:28.465678 13754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:11:28.468437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 13686
I20260926 13:11:28.468560 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:11:28.468739 392 external_mini_cluster.cc:1428] Setting key be324387c772f5dd69e3642b9b878a5c
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:28.475030 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.466335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.0.98.62:36785)
0926 13:11:28.466498 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.466506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.466633 (+ 127us) server_negotiation.cc:407] Connection header received
0926 13:11:28.467120 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.467122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.467138 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.467217 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.467739 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.468589 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.469419 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.469794 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.469888 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.471822 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.471831 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.471831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.471838 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.473484 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.473823 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.473826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.473827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.473957 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.474153 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.474154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.474155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.474432 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.474508 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.474702 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.474871 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260926 13:11:28.475031 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.466318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36785 (local address 127.0.0.1:43716)
0926 13:11:28.466476 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.466520 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:28.466990 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:28.467230 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:28.467232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:28.467475 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:28.467670 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.467674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.468674 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.468674 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.469348 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.469350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.469865 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.469866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.469930 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.470403 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:28.470410 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:28.471742 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:28.473560 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.473566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.473568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.473748 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.474030 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.474030 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.474030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.474100 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.474531 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:28.474534 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:28.474588 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 13:11:28.474656 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 13:11:28.474780 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260926 13:11:28.475889 13754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260926 13:11:28.476195 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.476100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:34538)
0926 13:11:28.476130 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.476173 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 13:11:28.476312 13704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
I20260926 13:11:28.476676 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.476589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:34552)
0926 13:11:28.476631 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.476655 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 13:11:28.476778 13754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41181: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 13:11:28.494309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41181
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36785,127.0.98.61:41181,127.0.98.60:37803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:11:28.526024 13754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } attempt: 1
I20260926 13:11:28.526504 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.526383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:34568)
0926 13:11:28.526436 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.526469 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 13:11:28.526954 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.526872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:34572)
0926 13:11:28.526908 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.526934 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 13:11:28.527050 13754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41181: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
W20260926 13:11:28.575356 13761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:28.575573 13761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:28.575604 13761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:28.576762 13761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:11:28.576802 13761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:28.576818 13761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:28.576834 13761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:11:28.576849 13761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:11:28.578337 13761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36785,127.0.98.61:41181,127.0.98.60:37803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:28.578594 13761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:28.578796 13761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:28.583549 13767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:28.583630 13766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:28.583621 13769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:28.583751 13761 server_base.cc:1034] running on GCE node
I20260926 13:11:28.584043 13761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:28.584295 13761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:28.585438 13761 hybrid_clock.cc:648] HybridClock initialized: now 1790428288585420 us; error 32 us; skew 500 ppm
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:28.588042 13761 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 13:11:28.588470 13761 webserver.cc:460] Webserver started at http://127.0.98.61:43917/ using document root <none> and password file <none>
I20260926 13:11:28.588721 13761 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:28.588771 13761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:28.588910 13761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:11:28.590018 13761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/data/instance:
uuid: "0ce4de4a3a084ef6b2c380b74f6e581c"
format_stamp: "Formatted at 2026-09-26 13:11:28 on dist-test-slave-2dw5"
server_key: "fc6b8901b99a97aab55a36ec7afff361"
server_key_iv: "2d6b319f5b33b9fc99beb19c9cb5c289"
server_key_version: "encryptionkey@0"
I20260926 13:11:28.590354 13761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/wal/instance:
uuid: "0ce4de4a3a084ef6b2c380b74f6e581c"
format_stamp: "Formatted at 2026-09-26 13:11:28 on dist-test-slave-2dw5"
server_key: "fc6b8901b99a97aab55a36ec7afff361"
server_key_iv: "2d6b319f5b33b9fc99beb19c9cb5c289"
server_key_version: "encryptionkey@0"
I20260926 13:11:28.591732 13761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:11:28.592427 13776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:28.592592 13761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:11:28.592679 13761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/wal
uuid: "0ce4de4a3a084ef6b2c380b74f6e581c"
format_stamp: "Formatted at 2026-09-26 13:11:28 on dist-test-slave-2dw5"
server_key: "fc6b8901b99a97aab55a36ec7afff361"
server_key_iv: "2d6b319f5b33b9fc99beb19c9cb5c289"
server_key_version: "encryptionkey@0"
I20260926 13:11:28.592739 13761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:28.595185 13754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } attempt: 2
I20260926 13:11:28.595649 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.595451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:34586)
0926 13:11:28.595567 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.595610 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 13:11:28.596078 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.595959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:34602)
0926 13:11:28.596017 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.596053 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 13:11:28.596240 13754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41181: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41181: connect: Connection refused (error 111)
I20260926 13:11:28.602706 13761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:28.604835 13761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:28.604966 13761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:28.608685 13761 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41181
I20260926 13:11:28.608718 13828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41181 every 8 connection(s)
I20260926 13:11:28.609064 13761 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
I20260926 13:11:28.609938 13829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:28.611639 13829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:28.619153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 13761
I20260926 13:11:28.619248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-1/wal/instance
I20260926 13:11:28.619415 392 external_mini_cluster.cc:1428] Setting key d641a32b93b0bd809f701cc650d5d94b
I20260926 13:11:28.620493 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.612326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.0.98.62:36785)
0926 13:11:28.612476 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.612478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.612613 (+ 135us) server_negotiation.cc:407] Connection header received
0926 13:11:28.613049 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.613050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.613062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.613124 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.613755 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.614256 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.615297 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.615533 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.615598 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.617798 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.617800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.617800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.617807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.619136 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.619571 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.619573 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.619573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.619630 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.619866 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.619867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.619867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.619985 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.620040 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.620304 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.620404 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260926 13:11:28.620675 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.612307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36785 (local address 127.0.0.1:43728)
0926 13:11:28.612466 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.612507 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:28.612942 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:28.613189 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:28.613192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:28.613439 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:28.613653 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.613658 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.614331 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.614331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.615216 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.615218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.615603 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.615604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.615703 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.616201 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:28.616211 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:28.617699 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:28.619214 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.619222 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.619225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.619508 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.619686 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.619686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.619687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.619809 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.620039 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:28.620046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:28.620190 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 13:11:28.620278 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 13:11:28.620443 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260926 13:11:28.621265 13829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 13:11:28.628942 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.621527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:34610)
0926 13:11:28.621569 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.621580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:28.621617 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:28.621846 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:28.621854 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:28.621985 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:28.622032 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.622033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.622985 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.622986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.623482 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.623483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.623920 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.623921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.623981 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.624451 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:28.624456 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:28.625779 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:28.627702 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.627703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.627704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.627864 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.628183 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.628184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.628184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.628246 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.628680 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:28.628681 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:28.628724 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 13:11:28.628782 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:11:28.628867 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 13:11:28.629115 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.621592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.0.98.61:41181)
0926 13:11:28.621726 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.621731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.621748 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:11:28.621766 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.621768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.621780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.621835 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.622094 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.622921 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.623549 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.623858 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.623917 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.625880 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.625891 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.625892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.625898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.627618 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.627990 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.627993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.627994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.628055 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.628338 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.628339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.628339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.628577 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.628648 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.628849 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.628922 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 13:11:28.629802 13829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:11:28.630299 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.630161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45034)
0926 13:11:28.630216 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.630274 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 13:11:28.630438 13779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
I20260926 13:11:28.630870 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.630768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45050)
0926 13:11:28.630802 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.630829 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 13:11:28.630961 13829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37803: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 13:11:28.646636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37803
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36785,127.0.98.61:41181,127.0.98.60:37803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:11:28.680199 13829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } attempt: 1
I20260926 13:11:28.680697 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.680538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45060)
0926 13:11:28.680608 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.680658 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 13:11:28.681124 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.681039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45076)
0926 13:11:28.681076 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.681104 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 13:11:28.681243 13829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37803: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
I20260926 13:11:28.687427 13754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } attempt: 3
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:11:28.694568 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.687794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:34618)
0926 13:11:28.687848 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.687862 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:28.687953 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:28.688131 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:28.688133 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:28.688276 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:28.688344 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.688345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.689300 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.689301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.689824 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.689825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.690281 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.690281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.690338 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.690766 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:28.690770 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:28.692170 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:28.693737 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.693737 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.693738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.693891 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.694064 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.694064 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.694064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.694112 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.694339 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:28.694340 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:28.694378 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 13:11:28.694422 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 13:11:28.694497 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 13:11:28.694612 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.687886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.0.98.61:41181)
0926 13:11:28.687940 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.687942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.687954 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:11:28.688007 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.688008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.688021 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.688097 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.688424 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.689223 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.689897 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.690213 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.690286 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.692254 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.692255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.692256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.692263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.693658 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.693952 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.693953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.693954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.694010 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.694157 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.694158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.694158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.694289 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.694348 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.694445 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.694525 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:11:28.695030 13754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:11:28.695364 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.695253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45082)
0926 13:11:28.695301 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.695338 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 13:11:28.695611 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.695555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45088)
0926 13:11:28.695578 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.695598 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 13:11:28.695686 13754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37803: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
I20260926 13:11:28.726884 13754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } attempt: 1
I20260926 13:11:28.727370 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.727249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45100)
0926 13:11:28.727305 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.727338 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 13:11:28.727809 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.727695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45108)
0926 13:11:28.727743 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.727790 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 13:11:28.727950 13754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37803: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
W20260926 13:11:28.730556 13836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:28.730751 13836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:28.730772 13836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:28.731967 13836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:11:28.732009 13836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:28.732021 13836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:28.732033 13836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:11:28.732043 13836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:11:28.733654 13836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36785,127.0.98.61:41181,127.0.98.60:37803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:28.733868 13836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:28.734059 13836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:28.739135 13841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:28.739135 13844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:28.739269 13842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:28.739588 13836 server_base.cc:1034] running on GCE node
I20260926 13:11:28.739811 13836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:28.740118 13836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:28.741295 13836 hybrid_clock.cc:648] HybridClock initialized: now 1790428288741272 us; error 35 us; skew 500 ppm
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:28.743903 13836 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 13:11:28.744347 13836 webserver.cc:460] Webserver started at http://127.0.98.60:34727/ using document root <none> and password file <none>
I20260926 13:11:28.744563 13836 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:28.744609 13836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:28.744731 13836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:11:28.745875 13836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/data/instance:
uuid: "ead5e93c060f44ad9273cd083f71da5e"
format_stamp: "Formatted at 2026-09-26 13:11:28 on dist-test-slave-2dw5"
server_key: "79409fa77d519ec80d6afae0d9e807f7"
server_key_iv: "b02852a513997dab784341375354f13d"
server_key_version: "encryptionkey@0"
I20260926 13:11:28.746208 13836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/wal/instance:
uuid: "ead5e93c060f44ad9273cd083f71da5e"
format_stamp: "Formatted at 2026-09-26 13:11:28 on dist-test-slave-2dw5"
server_key: "79409fa77d519ec80d6afae0d9e807f7"
server_key_iv: "b02852a513997dab784341375354f13d"
server_key_version: "encryptionkey@0"
I20260926 13:11:28.747586 13836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 13:11:28.748355 13851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:28.748539 13836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:28.748626 13836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/wal
uuid: "ead5e93c060f44ad9273cd083f71da5e"
format_stamp: "Formatted at 2026-09-26 13:11:28 on dist-test-slave-2dw5"
server_key: "79409fa77d519ec80d6afae0d9e807f7"
server_key_iv: "b02852a513997dab784341375354f13d"
server_key_version: "encryptionkey@0"
I20260926 13:11:28.748692 13836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:28.749413 13829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } attempt: 2
I20260926 13:11:28.749815 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.749680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45116)
0926 13:11:28.749737 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.749780 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 13:11:28.750165 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.750091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45122)
0926 13:11:28.750124 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.750148 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 13:11:28.750285 13829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37803: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37803: connect: Connection refused (error 111)
I20260926 13:11:28.764396 13836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:28.766573 13836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:28.766706 13836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:28.770591 13836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37803
I20260926 13:11:28.770668 13903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37803 every 8 connection(s)
I20260926 13:11:28.771018 13836 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
I20260926 13:11:28.771385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 13836
I20260926 13:11:28.771484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428136114168-392-0/minicluster-data/master-2/wal/instance
I20260926 13:11:28.771694 392 external_mini_cluster.cc:1428] Setting key 536ab58d577bb4e22740d0caf3c22ddd
I20260926 13:11:28.772167 13904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:28.774439 13904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:28.782408 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.775154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.0.98.62:36785)
0926 13:11:28.775214 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.775216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.775341 (+ 125us) server_negotiation.cc:407] Connection header received
0926 13:11:28.775392 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.775394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.775405 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.775466 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.775935 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.776383 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.777303 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.777593 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.777668 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.779584 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.779586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.779586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.779594 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.781152 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.781584 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.781585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.781585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.781644 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.781820 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.781821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.781821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.781955 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.782018 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.782239 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.782317 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:11:28.782459 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.772191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.0.98.60:37803)
0926 13:11:28.772373 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.772382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.772410 (+ 28us) server_negotiation.cc:407] Connection header received
0926 13:11:28.772850 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.772857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.772906 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.773033 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.773920 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.775000 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.775927 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.776370 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.776464 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.778887 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.778896 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.778896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.778903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.780759 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.781121 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.781124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.781124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.781230 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.781458 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.781459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.781459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.781783 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.781928 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.782030 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.782148 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260926 13:11:28.782466 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.775093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36785 (local address 127.0.0.1:43734)
0926 13:11:28.775252 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.775289 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:28.775349 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:28.775519 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:28.775521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:28.775777 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:28.775858 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.775859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.776463 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.776464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.777221 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.777224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.777672 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.777672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.777723 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.778128 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:28.778136 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:28.779499 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:28.781230 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.781237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.781239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.781500 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.781713 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.781714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.781714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.781764 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.782020 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:28.782027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:28.782080 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 13:11:28.782192 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 13:11:28.782307 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260926 13:11:28.783080 13904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:28.789856 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.783432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.0.98.61:41181)
0926 13:11:28.783477 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.783479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.783486 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:11:28.783506 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.783507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.783515 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.783554 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.783809 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.784350 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.785028 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.785234 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.785288 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.787041 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.787042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.787043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.787050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.788828 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.789148 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.789149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.789149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.789209 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.789368 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.789368 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.789369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.789478 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.789526 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.789705 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.789788 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:11:28.789878 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.783328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41181 (local address 127.0.0.1:34632)
0926 13:11:28.783413 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.783421 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:28.783449 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:28.783556 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:28.783557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:28.783691 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:28.783735 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.783736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.784461 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.784462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.784957 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.784958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.785292 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.785293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.785347 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.785724 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:28.785729 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:28.786957 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:28.788915 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.788915 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.788916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.789077 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.789265 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.789266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.789266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.789317 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.789553 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:28.789553 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:28.789606 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 13:11:28.789671 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:11:28.789778 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 13:11:28.790247 13904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:11:28.790572 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.783342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.0.98.62:36785)
0926 13:11:28.783377 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.783379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.783385 (+ 6us) server_negotiation.cc:407] Connection header received
0926 13:11:28.783405 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.783406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.783415 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.783598 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.784088 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.784544 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.785171 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.785366 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.785418 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.787659 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.787661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.787661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.787668 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.789200 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.789630 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.789631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.789631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.789680 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.789870 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.789871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.789872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.790125 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.790317 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.790356 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.790433 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:11:28.796576 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.790490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45150)
0926 13:11:28.790525 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.790535 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:28.790571 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:28.790694 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:28.790695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:28.790810 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:28.790903 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.790905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.791647 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.791648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.792171 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.792173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.792498 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.792498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.792553 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.792946 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:28.792949 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:28.794275 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:28.795707 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.795708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.795708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.795849 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.795996 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.795997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.795997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.796056 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.796276 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:28.796277 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:28.796314 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 13:11:28.796373 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:11:28.796470 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 13:11:28.796623 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.790525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45150 (local address 127.0.98.60:37803)
0926 13:11:28.790556 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.790558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.790565 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:11:28.790612 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.790613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.790622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.790662 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.790978 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.791577 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.792240 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.792439 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.792485 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.794360 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.794362 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.794362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.794370 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.795642 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.795903 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.795904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.795904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.795947 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.796122 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.796123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.796124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.796222 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.796274 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.796395 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.796517 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":18}
I20260926 13:11:28.797812 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.791157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.0.98.61:41181)
0926 13:11:28.791191 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.791193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.791200 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:11:28.791218 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.791219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.791227 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.791285 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.791686 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.792105 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.792832 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.793080 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.793129 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.795324 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.795326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.795326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.795333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.796696 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.797121 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.797121 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.797122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.797171 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.797360 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.797361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.797361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.797468 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.797628 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.797662 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.797729 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:11:28.798120 13904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e: Bootstrap starting.
I20260926 13:11:28.799175 13904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:28.799291 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:11:28.799460 13904 log.cc:824] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:28.800222 13904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e: No bootstrap required, opened a new log
I20260926 13:11:28.802428 13904 raft_consensus.cc:348] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } }
I20260926 13:11:28.802736 13904 raft_consensus.cc:374] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:28.802831 13904 raft_consensus.cc:729] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ead5e93c060f44ad9273cd083f71da5e, State: Initialized, Role: FOLLOWER
I20260926 13:11:28.803068 13904 consensus_queue.cc:260] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } }
I20260926 13:11:28.803088 13754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } attempt: 2
I20260926 13:11:28.803722 13904 sys_catalog.cc:564] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [sys.catalog]: configured and running, proceeding with master startup.
W20260926 13:11:28.808267 13929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:11:28.808357 13929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:11:28.805797 13836 master_runner.cc:186] Error getting master registration for 127.0.98.62:36785: OK, instance_id { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" instance_seqno: 1790428288461843 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:11:28.808635 13836 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:11:28.805286 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } } }
I20260926 13:11:28.808947 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:11:28.809089 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.800725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.0.98.62:36785)
0926 13:11:28.800801 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.800802 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.800810 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:11:28.800847 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.800848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.800859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.800918 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.801686 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.802402 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.803374 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.803637 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.803676 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.805392 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.805394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.805395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.805402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.807561 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.807891 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.807892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.807893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.807947 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.808345 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.808346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.808346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.808474 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.808581 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.808915 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.808990 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:11:28.810096 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.800681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34644 (local address 127.0.98.61:41181)
0926 13:11:28.800765 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.800767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.801035 (+ 268us) server_negotiation.cc:407] Connection header received
0926 13:11:28.801166 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.801167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.801177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.801206 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.801878 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.802596 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.803545 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.803846 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.803881 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.805353 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.805355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.805355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.805362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.806806 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.807228 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.807229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.807229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.807281 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.808945 (+ 1664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.808946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.808946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.809084 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.809153 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.809928 (+ 775us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.810008 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:11:28.811511 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.800825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.0.98.60:37803)
0926 13:11:28.800873 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.800875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.801035 (+ 160us) server_negotiation.cc:407] Connection header received
0926 13:11:28.801126 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.801127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.801135 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.801166 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.801871 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.802600 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.804414 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.804683 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.804724 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.806604 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.806606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.806606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.806613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.810032 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.810392 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.810393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.810393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.810450 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.810642 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.810643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.810644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.810792 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.810889 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.811362 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.811455 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:11:28.813575 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.803318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45176)
0926 13:11:28.803374 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.803385 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:28.803420 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:28.804220 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:28.804221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:28.804391 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:28.804432 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.804433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.807464 (+ 3031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.807466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.808214 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.808216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.808672 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.808673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.808729 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.809206 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:28.809211 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:28.810662 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:28.812648 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.812649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.812650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.812806 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.813069 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.813070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.813070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.813116 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.813356 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:28.813356 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:28.813391 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 13:11:28.813433 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 13:11:28.813503 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 13:11:28.813663 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.803482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.0.98.60:37803)
0926 13:11:28.804026 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.804028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.804049 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:11:28.804068 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.804068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.804077 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.804203 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.805668 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.807388 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.808305 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.808591 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.808866 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.810769 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.810770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.810770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.810777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.812561 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.812918 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.812919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.812920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.812998 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.813171 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.813172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.813172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.813296 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.813357 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.813468 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.813545 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":68,"threads_started":1}
I20260926 13:11:28.814903 13754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced: Bootstrap starting.
I20260926 13:11:28.815665 13754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:28.815986 13754 log.cc:824] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:28.816744 13754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced: No bootstrap required, opened a new log
I20260926 13:11:28.818161 13754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } }
I20260926 13:11:28.818303 13754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:28.818336 13754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fe39829a6f4561a6b7f7e3c6749ced, State: Initialized, Role: FOLLOWER
I20260926 13:11:28.818454 13754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } }
I20260926 13:11:28.818687 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } } }
I20260926 13:11:28.818761 13754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:11:28.818768 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [sys.catalog]: This master's current role is: FOLLOWER
W20260926 13:11:28.820477 13943 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:11:28.820585 13943 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:11:28.821343 13686 master_runner.cc:186] Error getting master registration for 127.0.98.61:41181: OK, instance_id { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" instance_seqno: 1790428288607601 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:11:28.824292 13686 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:11:28.837363 13932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:11:28.837515 13932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } }
I20260926 13:11:28.837807 13932 leader_election.cc:288] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ce4de4a3a084ef6b2c380b74f6e581c (127.0.98.61:41181), ead5e93c060f44ad9273cd083f71da5e (127.0.98.60:37803)
I20260926 13:11:28.838181 13878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead5e93c060f44ad9273cd083f71da5e" is_pre_election: true
I20260926 13:11:28.838349 13878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65fe39829a6f4561a6b7f7e3c6749ced in term 0.
I20260926 13:11:28.838563 13704 leader_election.cc:302] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65fe39829a6f4561a6b7f7e3c6749ced, ead5e93c060f44ad9273cd083f71da5e; no voters:
I20260926 13:11:28.838707 13932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:11:28.838770 13932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:11:28.838794 13932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:28.839859 13932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } }
I20260926 13:11:28.840059 13932 leader_election.cc:288] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [CANDIDATE]: Term 1 election: Requested vote from peers 0ce4de4a3a084ef6b2c380b74f6e581c (127.0.98.61:41181), ead5e93c060f44ad9273cd083f71da5e (127.0.98.60:37803)
I20260926 13:11:28.840329 13878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead5e93c060f44ad9273cd083f71da5e"
I20260926 13:11:28.840427 13878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:28.841313 13878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65fe39829a6f4561a6b7f7e3c6749ced in term 1.
I20260926 13:11:28.841507 13704 leader_election.cc:302] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65fe39829a6f4561a6b7f7e3c6749ced, ead5e93c060f44ad9273cd083f71da5e; no voters:
I20260926 13:11:28.841508 13829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } attempt: 3
I20260926 13:11:28.841699 13932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:11:28.841887 13932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [term 1 LEADER]: Becoming Leader. State: Replica: 65fe39829a6f4561a6b7f7e3c6749ced, State: Running, Role: LEADER
I20260926 13:11:28.841998 13932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } }
I20260926 13:11:28.842480 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65fe39829a6f4561a6b7f7e3c6749ced. Latest consensus state: current_term: 1 leader_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } } }
I20260926 13:11:28.842554 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:28.843291 13947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:11:28.843693 13947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 13:11:28 dist-test-slave-2dw5 krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:11:28.847114 13878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [term 1 FOLLOWER]: Refusing update from remote peer 65fe39829a6f4561a6b7f7e3c6749ced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:11:28.847621 13932 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:11:28.849349 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65fe39829a6f4561a6b7f7e3c6749ced. Latest consensus state: current_term: 1 leader_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } } }
I20260926 13:11:28.849447 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:11:28.850474 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.841759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37803 (local address 127.0.0.1:45192)
0926 13:11:28.841849 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:28.841860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:28.841901 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:28.842145 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:28.842146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:28.842327 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:28.842390 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.842393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.843309 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.843311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.844060 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:28.844063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.844582 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.844583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:28.844639 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.845113 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:28.845119 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:28.846630 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:28.849030 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.849032 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.849033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.849267 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.849687 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:28.849688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:28.849688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:28.849784 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:28.850141 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:28.850142 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:28.850208 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 13:11:28.850270 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:11:28.850381 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 13:11:28.850828 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } } }
I20260926 13:11:28.850934 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:28.851157 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:28.841975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.0.98.60:37803)
0926 13:11:28.842015 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:28.842017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:28.842024 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:11:28.842045 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:28.842045 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:28.842056 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:28.842098 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:28.842485 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.843221 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.844153 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:28.844459 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:28.844521 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:28.846786 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:28.846788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:28.846788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:28.846797 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:28.848918 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.849366 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.849367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.849367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.849447 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:28.849880 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:28.849881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:28.849882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:28.850038 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:28.850098 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:28.850959 (+ 861us) server_negotiation.cc:299] Negotiation successful
0926 13:11:28.851066 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:11:28.851329 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } } }
I20260926 13:11:28.851416 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:11:28.851611 13948 mvcc.cc:205] Tried to move back new op lower bound from 7333594271114006528 to 7333594271097331712. Current Snapshot: MvccSnapshot[applied={T|T < 7333594271114006528}]
I20260926 13:11:28.852442 13949 mvcc.cc:205] Tried to move back new op lower bound from 7333594271114006528 to 7333594271097331712. Current Snapshot: MvccSnapshot[applied={T|T < 7333594271114006528}]
I20260926 13:11:28.853603 13947 catalog_manager.cc:1347] Generated new cluster ID: 98767a8026a04f678eae5a41ef265296
I20260926 13:11:28.853714 13947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:11:28.854508 13829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c: Bootstrap starting.
I20260926 13:11:28.855316 13829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:28.855623 13829 log.cc:824] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:28.867955 13829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c: No bootstrap required, opened a new log
I20260926 13:11:28.869750 13947 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:11:28.870272 13829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } }
I20260926 13:11:28.870452 13829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:28.870496 13829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce4de4a3a084ef6b2c380b74f6e581c, State: Initialized, Role: FOLLOWER
I20260926 13:11:28.870702 13829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } }
I20260926 13:11:28.871160 13829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:11:28.871203 13802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c"
I20260926 13:11:28.871295 13802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:28.872215 13802 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65fe39829a6f4561a6b7f7e3c6749ced in term 1.
I20260926 13:11:28.872362 13803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" is_pre_election: true
I20260926 13:11:28.872524 13801 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [term 1 FOLLOWER]: Refusing update from remote peer 65fe39829a6f4561a6b7f7e3c6749ced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:11:28.873317 13947 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:11:28.873325 13945 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:11:28.878206 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } } }
I20260926 13:11:28.878405 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:11:28.878513 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65fe39829a6f4561a6b7f7e3c6749ced. Latest consensus state: current_term: 1 leader_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } } }
I20260926 13:11:28.878633 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:11:28.878700 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } } }
I20260926 13:11:28.878803 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:11:28.878928 13947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced: Generated new TSK 0
I20260926 13:11:28.879320 13947 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:11:28.879709 13966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:11:28.879772 13966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:11:28.879974 13968 mvcc.cc:205] Tried to move back new op lower bound from 7333594271201914880 to 7333594271097331712. Current Snapshot: MvccSnapshot[applied={T|T < 7333594271114006528}]
I20260926 13:11:28.880788 13952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fe39829a6f4561a6b7f7e3c6749ced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36785 } } peers { permanent_uuid: "0ce4de4a3a084ef6b2c380b74f6e581c" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41181 } } peers { permanent_uuid: "ead5e93c060f44ad9273cd083f71da5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37803 } } }
I20260926 13:11:28.880889 13952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fe39829a6f4561a6b7f7e3c6749ced [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:29.808969 13929 catalog_manager.cc:1259] Loaded cluster ID: 98767a8026a04f678eae5a41ef265296
I20260926 13:11:29.809082 13929 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e: loading cluster ID for follower catalog manager: success
I20260926 13:11:29.810081 13929 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e: acquiring CA information for follower catalog manager: success
I20260926 13:11:29.810451 13929 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ead5e93c060f44ad9273cd083f71da5e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:11:29.880383 13966 catalog_manager.cc:1259] Loaded cluster ID: 98767a8026a04f678eae5a41ef265296
I20260926 13:11:29.880492 13966 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c: loading cluster ID for follower catalog manager: success
I20260926 13:11:29.881369 13966 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c: acquiring CA information for follower catalog manager: success
I20260926 13:11:29.881685 13966 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ce4de4a3a084ef6b2c380b74f6e581c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:11:29.942965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 13686
I20260926 13:11:29.947916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 13761
I20260926 13:11:29.952924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 13836
2026-09-26T13:11:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2702 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:11:30 dist-test-slave-2dw5 krb5kdc[13981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:11:30 dist-test-slave-2dw5 krb5kdc[13981](info): set up 2 sockets
Sep 26 13:11:30 dist-test-slave-2dw5 krb5kdc[13981](info): commencing operation
krb5kdc: starting...
W20260926 13:11:30.914808 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:11:31 dist-test-slave-2dw5 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:11:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:11:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:11:31.042930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41291
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:31.124539 13994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:31.124786 13994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:31.124822 13994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:31.126049 13994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:11:31.126111 13994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:31.126135 13994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:31.126154 13994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:11:31.126173 13994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:11:31.127802 13994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:31.128077 13994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:31.128312 13994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:31.133366 14002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:31.133399 14000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:31.133472 13994 server_base.cc:1034] running on GCE node
W20260926 13:11:31.133368 13999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:31.133939 13994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:31.134224 13994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:31.135378 13994 hybrid_clock.cc:648] HybridClock initialized: now 1790428291135360 us; error 37 us; skew 500 ppm
Sep 26 13:11:31 dist-test-slave-2dw5 krb5kdc[13981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:31.137859 13994 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:11:31.138320 13994 webserver.cc:460] Webserver started at http://127.0.98.62:44761/ using document root <none> and password file <none>
I20260926 13:11:31.138515 13994 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:31.138561 13994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:31.138680 13994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:11:31.139784 13994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "2be06a558d184f0ba44bbdbcad5325a9"
format_stamp: "Formatted at 2026-09-26 13:11:31 on dist-test-slave-2dw5"
server_key: "74c2fe55a1ab9d345cb02c4e54082fba"
server_key_iv: "5a52e0995202e2a8a4d437a88ae12b0f"
server_key_version: "encryptionkey@0"
I20260926 13:11:31.140122 13994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "2be06a558d184f0ba44bbdbcad5325a9"
format_stamp: "Formatted at 2026-09-26 13:11:31 on dist-test-slave-2dw5"
server_key: "74c2fe55a1ab9d345cb02c4e54082fba"
server_key_iv: "5a52e0995202e2a8a4d437a88ae12b0f"
server_key_version: "encryptionkey@0"
I20260926 13:11:31.141358 13994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:11:31.142172 14009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:31.142320 13994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:11:31.142405 13994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "2be06a558d184f0ba44bbdbcad5325a9"
format_stamp: "Formatted at 2026-09-26 13:11:31 on dist-test-slave-2dw5"
server_key: "74c2fe55a1ab9d345cb02c4e54082fba"
server_key_iv: "5a52e0995202e2a8a4d437a88ae12b0f"
server_key_version: "encryptionkey@0"
I20260926 13:11:31.142472 13994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:31.152980 13994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:31.155162 13994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:31.155292 13994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:31.159319 13994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41291
I20260926 13:11:31.159411 14061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41291 every 8 connection(s)
I20260926 13:11:31.159737 13994 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:11:31.160583 14062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:31.162815 14062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9: Bootstrap starting.
I20260926 13:11:31.163404 14062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:31.163654 14062 log.cc:824] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:31.164314 14062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9: No bootstrap required, opened a new log
I20260926 13:11:31.165870 14062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be06a558d184f0ba44bbdbcad5325a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41291 } }
I20260926 13:11:31.166013 14062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:31.166036 14062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be06a558d184f0ba44bbdbcad5325a9, State: Initialized, Role: FOLLOWER
I20260926 13:11:31.166170 14062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be06a558d184f0ba44bbdbcad5325a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41291 } }
I20260926 13:11:31.166232 14062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:11:31.166253 14062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:11:31.166298 14062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:31.166806 14062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be06a558d184f0ba44bbdbcad5325a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41291 } }
I20260926 13:11:31.166924 14062 leader_election.cc:302] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2be06a558d184f0ba44bbdbcad5325a9; no voters:
I20260926 13:11:31.167089 14062 leader_election.cc:288] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:11:31.167122 14065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:11:31.167311 14065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [term 1 LEADER]: Becoming Leader. State: Replica: 2be06a558d184f0ba44bbdbcad5325a9, State: Running, Role: LEADER
I20260926 13:11:31.167384 14062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:11:31.167412 14065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be06a558d184f0ba44bbdbcad5325a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41291 } }
I20260926 13:11:31.167570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 13994
I20260926 13:11:31.167672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:11:31.167865 392 external_mini_cluster.cc:1428] Setting key 5ee8d47f8b81b71e769a06647e220590
I20260926 13:11:31.167867 14067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2be06a558d184f0ba44bbdbcad5325a9. Latest consensus state: current_term: 1 leader_uuid: "2be06a558d184f0ba44bbdbcad5325a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be06a558d184f0ba44bbdbcad5325a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41291 } } }
I20260926 13:11:31.167948 14067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:31.168130 14074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:11:31.168642 14074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:11:31.169046 14066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2be06a558d184f0ba44bbdbcad5325a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be06a558d184f0ba44bbdbcad5325a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41291 } } }
I20260926 13:11:31.169128 14066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:31.170187 14074 catalog_manager.cc:1347] Generated new cluster ID: 82b92b8709aa4742b62f3eee4be098f6
I20260926 13:11:31.170233 14074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:11:31 dist-test-slave-2dw5 krb5kdc[13981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:31.178982 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:31.168504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.0.98.62:41291)
0926 13:11:31.168710 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:31.168720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:31.169036 (+ 316us) server_negotiation.cc:407] Connection header received
0926 13:11:31.169238 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:31.169246 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:31.169308 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:31.169446 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:31.170322 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:31.171275 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:31.172004 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:31.172445 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:31.172548 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:31.175135 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:31.175139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:31.175139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:31.175152 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:31.177368 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:31.177738 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:31.177742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:31.177743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:31.177853 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:31.178042 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:31.178043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:31.178043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:31.178307 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:31.178469 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:31.178569 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 13:11:31.178717 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":52,"threads_started":1}
I20260926 13:11:31.180440 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:11:31.188802 14074 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:11:31.189381 14074 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:11:31.197140 14074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2be06a558d184f0ba44bbdbcad5325a9: Generated new TSK 0
I20260926 13:11:31.197355 14074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:11:31.234273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 13994
2026-09-26T13:11:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1278 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:11:31 dist-test-slave-2dw5 krb5kdc[14089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:11:31 dist-test-slave-2dw5 krb5kdc[14089](info): set up 2 sockets
Sep 26 13:11:31 dist-test-slave-2dw5 krb5kdc[14089](info): commencing operation
krb5kdc: starting...
W20260926 13:11:32.183071 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:11:32 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:11:32.302567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33361
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:32.384778 14102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:32.385015 14102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:32.385046 14102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:32.386237 14102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:11:32.386287 14102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:32.386305 14102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:32.386319 14102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:11:32.386335 14102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:11:32.387950 14102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:32.388216 14102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:32.388433 14102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:32.393407 14107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:32.393435 14108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:32.393507 14102 server_base.cc:1034] running on GCE node
W20260926 13:11:32.393491 14110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:32.393903 14102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:32.394222 14102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:32.395377 14102 hybrid_clock.cc:648] HybridClock initialized: now 1790428292395348 us; error 47 us; skew 500 ppm
Sep 26 13:11:32 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:32.397976 14102 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:11:32.398463 14102 webserver.cc:460] Webserver started at http://127.0.98.62:42521/ using document root <none> and password file <none>
I20260926 13:11:32.398671 14102 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:32.398720 14102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:32.398862 14102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:11:32.399940 14102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "800ea0684eca4979b8005059382fdf42"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "1c9f21c92e4ad83b92a66c496ed356ff"
server_key_iv: "f31f5779eff6f5d084d3da46044b0578"
server_key_version: "encryptionkey@0"
I20260926 13:11:32.400281 14102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "800ea0684eca4979b8005059382fdf42"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "1c9f21c92e4ad83b92a66c496ed356ff"
server_key_iv: "f31f5779eff6f5d084d3da46044b0578"
server_key_version: "encryptionkey@0"
I20260926 13:11:32.401571 14102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:11:32.402313 14117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:32.402472 14102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:11:32.402566 14102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "800ea0684eca4979b8005059382fdf42"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "1c9f21c92e4ad83b92a66c496ed356ff"
server_key_iv: "f31f5779eff6f5d084d3da46044b0578"
server_key_version: "encryptionkey@0"
I20260926 13:11:32.402630 14102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:32.422103 14102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:32.424140 14102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:32.424288 14102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:32.428277 14102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33361
I20260926 13:11:32.428318 14169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33361 every 8 connection(s)
I20260926 13:11:32.428678 14102 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:11:32.429546 14170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:32.432114 14170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Bootstrap starting.
I20260926 13:11:32.432739 14170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:32.433014 14170 log.cc:824] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:32.433802 14170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: No bootstrap required, opened a new log
I20260926 13:11:32.435691 14170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260926 13:11:32.435856 14170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:32.435879 14170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800ea0684eca4979b8005059382fdf42, State: Initialized, Role: FOLLOWER
I20260926 13:11:32.436029 14170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260926 13:11:32.436092 14170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:11:32.436115 14170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:11:32.436159 14170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:32.437048 14170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260926 13:11:32.437147 14170 leader_election.cc:302] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 800ea0684eca4979b8005059382fdf42; no voters:
I20260926 13:11:32.437304 14170 leader_election.cc:288] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:11:32.437384 14173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:11:32.437525 14173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 1 LEADER]: Becoming Leader. State: Replica: 800ea0684eca4979b8005059382fdf42, State: Running, Role: LEADER
I20260926 13:11:32.437556 14170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:11:32.437700 14173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260926 13:11:32.437817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 14102
I20260926 13:11:32.437929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:11:32.438143 392 external_mini_cluster.cc:1428] Setting key 36b50be30460f211b88c466344f97cd5
I20260926 13:11:32.438225 14174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "800ea0684eca4979b8005059382fdf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } } }
I20260926 13:11:32.438287 14174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:32.438432 14175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800ea0684eca4979b8005059382fdf42. Latest consensus state: current_term: 1 leader_uuid: "800ea0684eca4979b8005059382fdf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } } }
I20260926 13:11:32.438503 14175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:32.438918 14182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:11:32.439522 14182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:11:32.440979 14182 catalog_manager.cc:1347] Generated new cluster ID: 5bcf8929a6ca4eca94008f2f84826c8e
I20260926 13:11:32.441031 14182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:11:32 dist-test-slave-2dw5 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:32.450150 14182 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:11:32.450191 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:32.438931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.0.98.62:33361)
0926 13:11:32.439234 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:32.439243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:32.439280 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:11:32.439417 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:32.439426 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:32.439498 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:32.439678 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:32.440429 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.441826 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.442933 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.443540 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.443630 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:32.446341 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:32.446346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:32.446346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:32.446360 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:32.448370 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:32.448761 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:32.448764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:32.448764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:32.448886 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:32.449079 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:32.449079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:32.449080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:32.449435 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:32.449614 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:32.449772 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 13:11:32.449926 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":67,"threads_started":1}
I20260926 13:11:32.450986 14182 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:11:32.458787 14182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Generated new TSK 0
I20260926 13:11:32.459033 14182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:11:32.486716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33361
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:32.570236 14196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:32.570469 14196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:32.570508 14196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:32.571740 14196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:32.571786 14196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:32.571823 14196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:11:32.573304 14196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:32.573540 14196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:32.573753 14196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:32.578812 14202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:32.578806 14201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:32.578806 14204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:32.579173 14196 server_base.cc:1034] running on GCE node
I20260926 13:11:32.579434 14196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:32.579676 14196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:32.580824 14196 hybrid_clock.cc:648] HybridClock initialized: now 1790428292580806 us; error 35 us; skew 500 ppm
Sep 26 13:11:32 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:32.583561 14196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:11:32.584091 14196 webserver.cc:460] Webserver started at http://127.0.98.1:45863/ using document root <none> and password file <none>
I20260926 13:11:32.584302 14196 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:32.584353 14196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:32.584481 14196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:11:32.585548 14196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "a552129a0c924f4a8e0fc407905eb409"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "a6964fe0cecbe31f4ef045b086de1c32"
server_key_iv: "e56ac9f699469a007ffcde15711be130"
server_key_version: "encryptionkey@0"
I20260926 13:11:32.585882 14196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a552129a0c924f4a8e0fc407905eb409"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "a6964fe0cecbe31f4ef045b086de1c32"
server_key_iv: "e56ac9f699469a007ffcde15711be130"
server_key_version: "encryptionkey@0"
I20260926 13:11:32.587312 14196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:11:32.588234 14211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:32.588428 14196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:32.588521 14196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "a552129a0c924f4a8e0fc407905eb409"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "a6964fe0cecbe31f4ef045b086de1c32"
server_key_iv: "e56ac9f699469a007ffcde15711be130"
server_key_version: "encryptionkey@0"
I20260926 13:11:32.588593 14196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:32.624490 14196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:32.626696 14196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:32.626868 14196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:32.627116 14196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:32.627441 14196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:32.627475 14196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:32.627504 14196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:32.627521 14196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:32.633149 14196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46411
I20260926 13:11:32.633208 14324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46411 every 8 connection(s)
I20260926 13:11:32.633528 14196 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:11:32 dist-test-slave-2dw5 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:32.642951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 14196
I20260926 13:11:32.643059 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:32.634236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47439 (local address 127.0.98.62:33361)
0926 13:11:32.634281 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:32.634283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:32.634491 (+ 208us) server_negotiation.cc:407] Connection header received
0926 13:11:32.635059 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:32.635060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:32.635072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:32.635124 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:32.636095 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.636610 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.637656 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.637860 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.637915 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:32.640186 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:32.640187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:32.640188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:32.640195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:32.641587 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:32.641971 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:32.641972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:32.641972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:32.642029 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:32.642283 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:32.642284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:32.642284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:32.642436 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:32.642496 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:32.642863 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 13:11:32.642963 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:11:32.643071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:11:32.643097 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:32.634219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:47439)
0926 13:11:32.634371 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:32.634407 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:32.634960 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:32.635197 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:32.635201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:32.635789 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:32.635994 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:32.636000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.636695 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.636697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:32.637550 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:32.637554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.637932 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.637933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:32.638064 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:32.638547 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:32.638555 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:32.640082 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:32.641663 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:32.641677 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:32.641680 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:32.641899 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:32.642085 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:32.642086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:32.642086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:32.642192 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:32.642502 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:32.642508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:32.642637 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 13:11:32.642723 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 13:11:32.642884 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260926 13:11:32.643297 392 external_mini_cluster.cc:1428] Setting key 8cbc65cae4e1c93564da6f9aacf43618
I20260926 13:11:32.643672 14325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33361
I20260926 13:11:32.643764 14325 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:32.643936 14325 heartbeater.cc:511] Master 127.0.98.62:33361 requested a full tablet report, sending...
I20260926 13:11:32.644467 14134 ts_manager.cc:194] Registered new tserver with Master: a552129a0c924f4a8e0fc407905eb409 (127.0.98.1:46411)
I20260926 13:11:32.645308 14134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47439
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:11:32.670567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33361
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:32.752202 14330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:32.752432 14330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:32.752468 14330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:32.753756 14330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:32.753816 14330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:32.753854 14330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:11:32.755519 14330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:32.755787 14330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:32.756012 14330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:11:32.761322 14330 server_base.cc:1034] running on GCE node
W20260926 13:11:32.761279 14338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:32.761283 14336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:32.761279 14335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:32.761713 14330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:32.762048 14330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:32.763258 14330 hybrid_clock.cc:648] HybridClock initialized: now 1790428292763242 us; error 34 us; skew 500 ppm
Sep 26 13:11:32 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:32.765769 14330 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:11:32.766247 14330 webserver.cc:460] Webserver started at http://127.0.98.2:33679/ using document root <none> and password file <none>
I20260926 13:11:32.766461 14330 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:32.766511 14330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:32.766618 14330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:11:32.767714 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "9e733b9d5c1b4615ae610b08f0995ba3"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "be537616b429c6b6e205b19ca3666d5a"
server_key_iv: "97815b5bbd01188f514eeb490f55666e"
server_key_version: "encryptionkey@0"
I20260926 13:11:32.768045 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9e733b9d5c1b4615ae610b08f0995ba3"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "be537616b429c6b6e205b19ca3666d5a"
server_key_iv: "97815b5bbd01188f514eeb490f55666e"
server_key_version: "encryptionkey@0"
I20260926 13:11:32.769292 14330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:11:32.769971 14345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:32.770120 14330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:11:32.770197 14330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "9e733b9d5c1b4615ae610b08f0995ba3"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "be537616b429c6b6e205b19ca3666d5a"
server_key_iv: "97815b5bbd01188f514eeb490f55666e"
server_key_version: "encryptionkey@0"
I20260926 13:11:32.770267 14330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:32.807981 14330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:32.810153 14330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:32.810285 14330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:32.810540 14330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:32.810874 14330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:32.810907 14330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:32.810930 14330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:32.810953 14330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:32.816475 14330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34727
I20260926 13:11:32.816529 14458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34727 every 8 connection(s)
I20260926 13:11:32.816849 14330 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:11:32 dist-test-slave-2dw5 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:32.825747 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:32.817523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49629 (local address 127.0.98.62:33361)
0926 13:11:32.817574 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:32.817576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:32.817751 (+ 175us) server_negotiation.cc:407] Connection header received
0926 13:11:32.818242 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:32.818243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:32.818253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:32.818314 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:32.819309 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.819816 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.820801 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.820991 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.821044 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:32.823187 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:32.823189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:32.823189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:32.823197 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:32.824424 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:32.824840 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:32.824841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:32.824841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:32.824894 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:32.825128 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:32.825129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:32.825129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:32.825289 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:32.825343 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:32.825602 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 13:11:32.825677 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:11:32.825917 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:32.817485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:49629)
0926 13:11:32.817627 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:32.817669 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:32.818158 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:32.818425 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:32.818428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:32.819033 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:32.819229 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:32.819233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.819917 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.819918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:32.820733 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:32.820735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.821054 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.821054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:32.821149 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:32.821608 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:32.821616 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:32.823082 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:32.824526 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:32.824534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:32.824536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:32.824770 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:32.824954 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:32.824955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:32.824955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:32.825066 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:32.825351 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:32.825357 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:32.825490 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 13:11:32.825579 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:11:32.825705 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260926 13:11:32.826169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 14330
I20260926 13:11:32.826251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:11:32.826428 392 external_mini_cluster.cc:1428] Setting key 94795c3c9e03ec9cc82f9bb6894c4770
I20260926 13:11:32.826535 14459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33361
I20260926 13:11:32.826948 14459 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:32.827173 14459 heartbeater.cc:511] Master 127.0.98.62:33361 requested a full tablet report, sending...
I20260926 13:11:32.827615 14134 ts_manager.cc:194] Registered new tserver with Master: 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727)
I20260926 13:11:32.828074 14134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49629
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:11:32.855142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33361
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:32.938159 14464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:32.938446 14464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:32.938484 14464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:32.939929 14464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:32.939980 14464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:32.940021 14464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:11:32.941532 14464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:32.941771 14464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:32.941993 14464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:32.947062 14472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:32.947153 14470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:32.947053 14469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:32.947275 14464 server_base.cc:1034] running on GCE node
I20260926 13:11:32.947474 14464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:32.947731 14464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:32.948887 14464 hybrid_clock.cc:648] HybridClock initialized: now 1790428292948870 us; error 30 us; skew 500 ppm
Sep 26 13:11:32 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:32.951620 14464 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:11:32.952111 14464 webserver.cc:460] Webserver started at http://127.0.98.3:33713/ using document root <none> and password file <none>
I20260926 13:11:32.952322 14464 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:32.952368 14464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:32.952481 14464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:11:32.953553 14464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "e96afd1c2edb456782394997a593285c"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "fff86af2c701de71752312b78415e21f"
server_key_iv: "3fbf4e1eaf2a0a1b8994222fea08c9fb"
server_key_version: "encryptionkey@0"
I20260926 13:11:32.953881 14464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e96afd1c2edb456782394997a593285c"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "fff86af2c701de71752312b78415e21f"
server_key_iv: "3fbf4e1eaf2a0a1b8994222fea08c9fb"
server_key_version: "encryptionkey@0"
I20260926 13:11:32.955225 14464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:11:32.956066 14479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:32.956236 14464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:11:32.956323 14464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "e96afd1c2edb456782394997a593285c"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "fff86af2c701de71752312b78415e21f"
server_key_iv: "3fbf4e1eaf2a0a1b8994222fea08c9fb"
server_key_version: "encryptionkey@0"
I20260926 13:11:32.956399 14464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:32.970197 14464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:32.972580 14464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:32.972702 14464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:32.972946 14464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:32.973248 14464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:32.973280 14464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:32.973304 14464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:32.973320 14464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:32.979380 14464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40303
I20260926 13:11:32.979447 14592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40303 every 8 connection(s)
I20260926 13:11:32.979744 14464 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:11:32.980122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 14464
I20260926 13:11:32.980191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:11:32.980348 392 external_mini_cluster.cc:1428] Setting key d5d240d8ed2bf45b5f09389dae3fc835
Sep 26 13:11:32 dist-test-slave-2dw5 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:32.988951 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:32.980529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43955 (local address 127.0.98.62:33361)
0926 13:11:32.980587 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:32.980589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:32.980832 (+ 243us) server_negotiation.cc:407] Connection header received
0926 13:11:32.981329 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:32.981331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:32.981341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:32.981392 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:32.982344 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.982867 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.983810 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.984013 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.984066 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:32.986358 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:32.986359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:32.986359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:32.986367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:32.987645 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:32.988086 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:32.988087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:32.988088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:32.988140 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:32.988346 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:32.988347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:32.988347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:32.988472 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:32.988526 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:32.988779 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 13:11:32.988860 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:11:32.989145 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:32.980536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:43955)
0926 13:11:32.980707 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:32.980752 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:32.981229 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:32.981467 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:32.981472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:32.982043 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:32.982240 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:32.982244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.982948 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.982950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:32.983729 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:32.983731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.984081 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.984082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:32.984196 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:32.984684 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:32.984693 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:32.986263 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:32.987719 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:32.987727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:32.987729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:32.988015 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:32.988192 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:32.988192 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:32.988193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:32.988292 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:32.988527 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:32.988534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:32.988665 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 13:11:32.988751 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 13:11:32.988924 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260926 13:11:32.989666 14593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33361
I20260926 13:11:32.989771 14593 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:32.989962 14593 heartbeater.cc:511] Master 127.0.98.62:33361 requested a full tablet report, sending...
I20260926 13:11:32.990446 14134 ts_manager.cc:194] Registered new tserver with Master: e96afd1c2edb456782394997a593285c (127.0.98.3:40303)
I20260926 13:11:32.990820 14134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43955
I20260926 13:11:32.991806 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:11:32.999213 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:32.993159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.0.98.62:33361)
0926 13:11:32.993251 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:32.993253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:32.993326 (+ 73us) server_negotiation.cc:407] Connection header received
0926 13:11:32.993363 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:32.993364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:32.993373 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:32.993425 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:32.994014 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.994490 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.995383 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:32.995585 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:32.995654 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:32.996874 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:32.996876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:32.996876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:32.996883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:32.998053 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:32.998433 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:32.998433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:32.998434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:32.998517 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:32.998673 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:32.998673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:32.998674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:32.998815 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:32.998919 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:32.999069 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 13:11:32.999134 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 13:11:33.000861 14134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:11:33.002480 14134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:11:33.009128 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.005112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34727 (local address 127.0.0.1:60240)
0926 13:11:33.005224 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.005232 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.005266 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.005709 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.005713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.005715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:33.005789 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.005792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.007218 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.007219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.008086 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.008089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.008800 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.008801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.008887 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.008902 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.008987 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.009036 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":6,"thread_start_us":67,"threads_started":1}
I20260926 13:11:33.009135 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.005381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.0.98.2:34727)
0926 13:11:33.005539 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.005543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.005551 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:11:33.005590 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.005593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.005605 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.005861 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:33.005941 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.007077 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.008179 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.008727 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.008792 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.008854 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.009005 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.009047 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260926 13:11:33.009312 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.005087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.0.98.3:40303)
0926 13:11:33.005633 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.005638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.005648 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:11:33.005701 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.005703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.005718 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.005786 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:33.005900 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.006673 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.008088 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.008759 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.009093 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.009161 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.009187 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.009231 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":98,"threads_started":1}
I20260926 13:11:33.009361 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.004840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40303 (local address 127.0.0.1:58780)
0926 13:11:33.005057 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.005068 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.005127 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.005784 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.005785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.005787 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:33.005836 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.005838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.006757 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.006758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.007999 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.008002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.008856 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.008857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.008934 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.008938 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.008968 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.009039 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":8,"thread_start_us":93,"threads_started":1}
I20260926 13:11:33.009671 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.004840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46411 (local address 127.0.0.1:47430)
0926 13:11:33.005023 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.005067 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.005130 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.005725 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.005726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.005727 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:33.005789 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.005791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.007151 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.007152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.008286 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.008288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.009191 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.009192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.009502 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.009506 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.009563 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.009602 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":48,"thread_start_us":128,"threads_started":1}
I20260926 13:11:33.009716 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.005294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.0.98.1:46411)
0926 13:11:33.005495 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.005500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.005510 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:11:33.005552 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.005554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.005570 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.005646 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:33.005864 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.007047 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.008369 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.009056 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.009112 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.009160 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.009593 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.009640 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
I20260926 13:11:33.010028 14392 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1ba8774a734282b181f7d0e946520e (DEFAULT_TABLE table=test-table [id=922bb57466c84597948977fae9f6f2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:11:33.010111 14527 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1ba8774a734282b181f7d0e946520e (DEFAULT_TABLE table=test-table [id=922bb57466c84597948977fae9f6f2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:11:33.010394 14527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1ba8774a734282b181f7d0e946520e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:33.010396 14392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1ba8774a734282b181f7d0e946520e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:33.010327 14259 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1ba8774a734282b181f7d0e946520e (DEFAULT_TABLE table=test-table [id=922bb57466c84597948977fae9f6f2e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:11:33.010627 14259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1ba8774a734282b181f7d0e946520e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:33.013164 14610 tablet_bootstrap.cc:492] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c: Bootstrap starting.
I20260926 13:11:33.013685 14612 tablet_bootstrap.cc:492] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Bootstrap starting.
I20260926 13:11:33.013908 14610 tablet_bootstrap.cc:654] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:33.014326 14612 tablet_bootstrap.cc:654] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:33.014621 14610 log.cc:824] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:33.014830 14612 log.cc:824] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:33.015048 14611 tablet_bootstrap.cc:492] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3: Bootstrap starting.
I20260926 13:11:33.015628 14611 tablet_bootstrap.cc:654] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:33.015666 14610 tablet_bootstrap.cc:492] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c: No bootstrap required, opened a new log
I20260926 13:11:33.015722 14610 ts_tablet_manager.cc:1397] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 13:11:33.016175 14611 log.cc:824] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:33.016734 14612 tablet_bootstrap.cc:492] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: No bootstrap required, opened a new log
I20260926 13:11:33.016826 14612 ts_tablet_manager.cc:1397] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 13:11:33.017000 14611 tablet_bootstrap.cc:492] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3: No bootstrap required, opened a new log
I20260926 13:11:33.017068 14611 ts_tablet_manager.cc:1397] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:11:33.017719 14610 raft_consensus.cc:348] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } }
I20260926 13:11:33.017927 14610 raft_consensus.cc:374] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:33.017975 14610 raft_consensus.cc:729] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e96afd1c2edb456782394997a593285c, State: Initialized, Role: FOLLOWER
I20260926 13:11:33.018146 14610 consensus_queue.cc:260] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } }
I20260926 13:11:33.018477 14610 ts_tablet_manager.cc:1428] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 13:11:33.018649 14593 heartbeater.cc:503] Master 127.0.98.62:33361 was elected leader, sending a full tablet report...
I20260926 13:11:33.018827 14612 raft_consensus.cc:348] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } }
I20260926 13:11:33.019043 14612 raft_consensus.cc:374] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:33.019091 14612 raft_consensus.cc:729] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552129a0c924f4a8e0fc407905eb409, State: Initialized, Role: FOLLOWER
I20260926 13:11:33.019222 14612 consensus_queue.cc:260] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } }
I20260926 13:11:33.019160 14611 raft_consensus.cc:348] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } }
I20260926 13:11:33.019322 14611 raft_consensus.cc:374] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:33.019361 14611 raft_consensus.cc:729] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e733b9d5c1b4615ae610b08f0995ba3, State: Initialized, Role: FOLLOWER
I20260926 13:11:33.019505 14612 ts_tablet_manager.cc:1428] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 13:11:33.019476 14611 consensus_queue.cc:260] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } }
I20260926 13:11:33.019683 14325 heartbeater.cc:503] Master 127.0.98.62:33361 was elected leader, sending a full tablet report...
I20260926 13:11:33.019776 14611 ts_tablet_manager.cc:1428] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 13:11:33.019809 14459 heartbeater.cc:503] Master 127.0.98.62:33361 was elected leader, sending a full tablet report...
W20260926 13:11:33.067427 14460 tablet.cc:2369] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:11:33.134272 14326 tablet.cc:2369] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:11:33.150897 14617 raft_consensus.cc:482] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:11:33.151044 14617 raft_consensus.cc:504] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } }
I20260926 13:11:33.151429 14617 leader_election.cc:288] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e96afd1c2edb456782394997a593285c (127.0.98.3:40303), 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727)
I20260926 13:11:33.154763 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.151707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37069 (local address 127.0.98.3:40303)
0926 13:11:33.151757 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.151759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.151885 (+ 126us) server_negotiation.cc:407] Connection header received
0926 13:11:33.151983 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.151984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.152006 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.152031 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:33.152290 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.152853 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.154041 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.154438 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.154495 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.154507 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.154676 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.154706 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:11:33.154806 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.151615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40303 (local address 127.0.98.1:37069)
0926 13:11:33.151820 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.151830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.151913 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.152115 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.152115 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.152117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:33.152197 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.152199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.152946 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.152948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.153937 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.153940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.154526 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.154527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.154602 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.154607 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.154672 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.154713 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":33,"thread_start_us":77,"threads_started":1}
I20260926 13:11:33.155226 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.151782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37251 (local address 127.0.98.2:34727)
0926 13:11:33.151857 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.151859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.151945 (+ 86us) server_negotiation.cc:407] Connection header received
0926 13:11:33.152064 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.152065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.152078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.152143 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:33.152395 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.153067 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.154267 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.154753 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.154812 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.154822 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.155128 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.155177 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:11:33.155233 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.151615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34727 (local address 127.0.98.1:37251)
0926 13:11:33.151879 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.151889 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.151949 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.152197 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.152198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.152200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:33.152297 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.152300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.153231 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.153232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.154016 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.154018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.154977 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.154978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.155053 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.155058 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.155110 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.155151 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":158,"thread_start_us":38,"threads_started":1}
I20260926 13:11:33.155305 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1ba8774a734282b181f7d0e946520e" candidate_uuid: "a552129a0c924f4a8e0fc407905eb409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e96afd1c2edb456782394997a593285c" is_pre_election: true
I20260926 13:11:33.155460 14547 raft_consensus.cc:2454] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a552129a0c924f4a8e0fc407905eb409 in term 0.
I20260926 13:11:33.155674 14213 leader_election.cc:302] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a552129a0c924f4a8e0fc407905eb409, e96afd1c2edb456782394997a593285c; no voters:
I20260926 13:11:33.155679 14413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1ba8774a734282b181f7d0e946520e" candidate_uuid: "a552129a0c924f4a8e0fc407905eb409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" is_pre_election: true
I20260926 13:11:33.155802 14413 raft_consensus.cc:2454] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a552129a0c924f4a8e0fc407905eb409 in term 0.
I20260926 13:11:33.155802 14617 raft_consensus.cc:2781] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:11:33.155846 14617 raft_consensus.cc:482] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:11:33.155867 14617 raft_consensus.cc:3037] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:33.156622 14617 raft_consensus.cc:504] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } }
I20260926 13:11:33.156757 14617 leader_election.cc:288] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 1 election: Requested vote from peers e96afd1c2edb456782394997a593285c (127.0.98.3:40303), 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727)
I20260926 13:11:33.156909 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1ba8774a734282b181f7d0e946520e" candidate_uuid: "a552129a0c924f4a8e0fc407905eb409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e96afd1c2edb456782394997a593285c"
I20260926 13:11:33.156985 14547 raft_consensus.cc:3037] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:33.156888 14413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1ba8774a734282b181f7d0e946520e" candidate_uuid: "a552129a0c924f4a8e0fc407905eb409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e733b9d5c1b4615ae610b08f0995ba3"
I20260926 13:11:33.157066 14413 raft_consensus.cc:3037] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:33.157716 14547 raft_consensus.cc:2454] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a552129a0c924f4a8e0fc407905eb409 in term 1.
I20260926 13:11:33.157845 14213 leader_election.cc:302] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a552129a0c924f4a8e0fc407905eb409, e96afd1c2edb456782394997a593285c; no voters:
I20260926 13:11:33.157859 14413 raft_consensus.cc:2454] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a552129a0c924f4a8e0fc407905eb409 in term 1.
I20260926 13:11:33.157925 14617 raft_consensus.cc:2781] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:11:33.158073 14617 raft_consensus.cc:686] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 1 LEADER]: Becoming Leader. State: Replica: a552129a0c924f4a8e0fc407905eb409, State: Running, Role: LEADER
I20260926 13:11:33.158169 14617 consensus_queue.cc:237] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } }
I20260926 13:11:33.159020 14132 catalog_manager.cc:5495] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 reported cstate change: term changed from 0 to 1, leader changed from <none> to a552129a0c924f4a8e0fc407905eb409 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a552129a0c924f4a8e0fc407905eb409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } health_report { overall_health: UNKNOWN } } }
I20260926 13:11:33.192845 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.190177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.0.98.1:46411)
0926 13:11:33.190233 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.190235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.190245 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:11:33.190287 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.190288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.190311 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.190379 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:11:33.190489 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.191080 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.192029 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.192310 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.192364 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.192528 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:11:33.192644 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:11:33.192718 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.192741 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.192764 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:11:33.195014 14547 raft_consensus.cc:1261] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c [term 1 FOLLOWER]: Refusing update from remote peer a552129a0c924f4a8e0fc407905eb409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:11:33.195012 14413 raft_consensus.cc:1261] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3 [term 1 FOLLOWER]: Refusing update from remote peer a552129a0c924f4a8e0fc407905eb409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:11:33.195429 14621 consensus_queue.cc:1035] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:11:33.195544 14617 consensus_queue.cc:1035] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:11:33.197856 14624 mvcc.cc:205] Tried to move back new op lower bound from 7333594288923590656 to 7333594288776429568. Current Snapshot: MvccSnapshot[applied={T|T < 7333594288923590656}]
I20260926 13:11:33.198191 14626 mvcc.cc:205] Tried to move back new op lower bound from 7333594288923590656 to 7333594288776429568. Current Snapshot: MvccSnapshot[applied={T|T < 7333594288923590656}]
I20260926 13:11:33.198320 14625 mvcc.cc:205] Tried to move back new op lower bound from 7333594288923590656 to 7333594288776429568. Current Snapshot: MvccSnapshot[applied={T|T < 7333594288923590656}]
I20260926 13:11:33.200888 14133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:11:33.201014 14133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:11:33.201804 14133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Sending DeleteTablet for 3 replicas of tablet 4b1ba8774a734282b181f7d0e946520e
I20260926 13:11:33.202366 14259 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1ba8774a734282b181f7d0e946520e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47430
I20260926 13:11:33.202381 14527 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1ba8774a734282b181f7d0e946520e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58780
I20260926 13:11:33.202391 14392 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1ba8774a734282b181f7d0e946520e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60240
I20260926 13:11:33.202705 14635 tablet_replica.cc:330] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3: stopping tablet replica
I20260926 13:11:33.202702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 14196
I20260926 13:11:33.202708 14634 tablet_replica.cc:330] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: stopping tablet replica
I20260926 13:11:33.203444 14635 raft_consensus.cc:2229] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:11:33.203646 14635 raft_consensus.cc:2258] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:11:33.203860 14636 tablet_replica.cc:330] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c: stopping tablet replica
I20260926 13:11:33.204077 14636 raft_consensus.cc:2229] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:11:33.204221 14636 raft_consensus.cc:2258] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:11:33.204787 14636 ts_tablet_manager.cc:1905] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:11:33.204797 14635 ts_tablet_manager.cc:1905] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:11:33.206547 14636 ts_tablet_manager.cc:1918] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:11:33.206629 14636 log.cc:1196] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/4b1ba8774a734282b181f7d0e946520e
I20260926 13:11:33.206643 14635 ts_tablet_manager.cc:1918] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:11:33.206756 14635 log.cc:1196] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/4b1ba8774a734282b181f7d0e946520e
I20260926 13:11:33.206949 14636 ts_tablet_manager.cc:1939] T 4b1ba8774a734282b181f7d0e946520e P e96afd1c2edb456782394997a593285c: Deleting consensus metadata
I20260926 13:11:33.207047 14635 ts_tablet_manager.cc:1939] T 4b1ba8774a734282b181f7d0e946520e P 9e733b9d5c1b4615ae610b08f0995ba3: Deleting consensus metadata
I20260926 13:11:33.207456 14120 catalog_manager.cc:4841] TS 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727): tablet 4b1ba8774a734282b181f7d0e946520e (table test-table [id=922bb57466c84597948977fae9f6f2e0]) successfully deleted
I20260926 13:11:33.207458 14119 catalog_manager.cc:4841] TS e96afd1c2edb456782394997a593285c (127.0.98.3:40303): tablet 4b1ba8774a734282b181f7d0e946520e (table test-table [id=922bb57466c84597948977fae9f6f2e0]) successfully deleted
W20260926 13:11:33.208462 14120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46411 (error 108)
I20260926 13:11:33.208704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 14330
I20260926 13:11:33.209281 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.208848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46411 (local address 127.0.0.1:47446)
0926 13:11:33.209204 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.209247 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334}
W20260926 13:11:33.209453 14120 catalog_manager.cc:4584] TS a552129a0c924f4a8e0fc407905eb409 (127.0.98.1:46411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b1ba8774a734282b181f7d0e946520e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46411: connect: Connection refused (error 111)
I20260926 13:11:33.213958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 14464
I20260926 13:11:33.218052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 14102
I20260926 13:11:33.222337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33361
--webserver_interface=127.0.98.62
--webserver_port=42521
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:33.302211 14638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:33.302426 14638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:33.302448 14638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:33.303779 14638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:11:33.303836 14638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:33.303850 14638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:33.303861 14638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:11:33.303879 14638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:11:33.305397 14638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:33.305617 14638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:33.305826 14638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:33.314383 14644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:33.314378 14647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:33.314353 14645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:33.314451 14638 server_base.cc:1034] running on GCE node
I20260926 13:11:33.314940 14638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:33.315248 14638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:33.316452 14638 hybrid_clock.cc:648] HybridClock initialized: now 1790428293316433 us; error 32 us; skew 500 ppm
Sep 26 13:11:33 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:33.319168 14638 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:11:33.319674 14638 webserver.cc:460] Webserver started at http://127.0.98.62:42521/ using document root <none> and password file <none>
I20260926 13:11:33.319887 14638 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:33.319952 14638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:33.321419 14638 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:11:33.322158 14654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:33.322350 14638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:11:33.322448 14638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "800ea0684eca4979b8005059382fdf42"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "1c9f21c92e4ad83b92a66c496ed356ff"
server_key_iv: "f31f5779eff6f5d084d3da46044b0578"
server_key_version: "encryptionkey@0"
I20260926 13:11:33.322897 14638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:33.346483 14638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:33.348667 14638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:33.348806 14638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:33.352807 14638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33361
I20260926 13:11:33.352847 14706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33361 every 8 connection(s)
I20260926 13:11:33.353309 14638 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:11:33.355008 14707 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:11:33.355669 14707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Bootstrap starting.
I20260926 13:11:33.357095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 14638
I20260926 13:11:33.357342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46411
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33361
--webserver_port=45863
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:11:33.364127 14707 log.cc:824] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:33.366005 14707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:11:33.366261 14707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Bootstrap complete.
I20260926 13:11:33.368417 14707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260926 13:11:33.368662 14707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800ea0684eca4979b8005059382fdf42, State: Initialized, Role: FOLLOWER
I20260926 13:11:33.368808 14707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260926 13:11:33.368886 14707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:11:33.368942 14707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:11:33.368988 14707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:11:33.370000 14707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260926 13:11:33.370095 14707 leader_election.cc:302] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 800ea0684eca4979b8005059382fdf42; no voters:
I20260926 13:11:33.370257 14707 leader_election.cc:288] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:11:33.370322 14711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:11:33.370509 14707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:11:33.370589 14711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 2 LEADER]: Becoming Leader. State: Replica: 800ea0684eca4979b8005059382fdf42, State: Running, Role: LEADER
I20260926 13:11:33.370941 14711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260926 13:11:33.371299 14711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800ea0684eca4979b8005059382fdf42. Latest consensus state: current_term: 2 leader_uuid: "800ea0684eca4979b8005059382fdf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } } }
I20260926 13:11:33.371371 14711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:33.371695 14712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "800ea0684eca4979b8005059382fdf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } } }
I20260926 13:11:33.371753 14712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:33.371848 14716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:11:33.372892 14716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:11:33.372994 14716 catalog_manager.cc:1259] Loaded cluster ID: 5bcf8929a6ca4eca94008f2f84826c8e
I20260926 13:11:33.373101 14716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:11:33.373683 14716 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 13:11:33.378463 14716 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 13:11:33.381101 14716 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:11:33.381316 14716 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Loaded TSK: 0
I20260926 13:11:33.381520 14716 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:11:33.444885 14709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:33.445123 14709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:33.445149 14709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:33.446372 14709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:33.446419 14709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:33.446450 14709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:11:33.448001 14709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45863
--tserver_master_addrs=127.0.98.62:33361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:33.448243 14709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:33.448470 14709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:33.453783 14737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:33.453760 14734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:33.453997 14709 server_base.cc:1034] running on GCE node
W20260926 13:11:33.454080 14735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:33.454324 14709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:33.454640 14709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:33.455822 14709 hybrid_clock.cc:648] HybridClock initialized: now 1790428293455798 us; error 50 us; skew 500 ppm
Sep 26 13:11:33 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:33.467545 14709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:11:33.468072 14709 webserver.cc:460] Webserver started at http://127.0.98.1:45863/ using document root <none> and password file <none>
I20260926 13:11:33.468302 14709 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:33.468356 14709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:33.469693 14709 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:33.470332 14744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:33.470616 14709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:33.470750 14709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "a552129a0c924f4a8e0fc407905eb409"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "a6964fe0cecbe31f4ef045b086de1c32"
server_key_iv: "e56ac9f699469a007ffcde15711be130"
server_key_version: "encryptionkey@0"
I20260926 13:11:33.471233 14709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:33.493253 14709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:33.495605 14709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:33.495751 14709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:33.495986 14709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:33.496423 14751 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:11:33.498474 14709 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:11:33.498539 14709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 13:11:33.498581 14709 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:11:33.499272 14709 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:11:33.499305 14709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 13:11:33.499378 14751 tablet_bootstrap.cc:492] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Bootstrap starting.
I20260926 13:11:33.506219 14709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46411
I20260926 13:11:33.506400 14858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46411 every 8 connection(s)
I20260926 13:11:33.506646 14709 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:11:33.509768 14751 log.cc:824] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:33.512244 14751 tablet_bootstrap.cc:492] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:11:33.512573 14751 tablet_bootstrap.cc:492] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Bootstrap complete.
I20260926 13:11:33.513129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 14709
I20260926 13:11:33.513367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34727
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33361
--webserver_port=33679
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:11:33.513437 14751 ts_tablet_manager.cc:1397] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
Sep 26 13:11:33 dist-test-slave-2dw5 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:33.514506 14751 raft_consensus.cc:348] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } }
I20260926 13:11:33.514833 14751 raft_consensus.cc:729] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552129a0c924f4a8e0fc407905eb409, State: Initialized, Role: FOLLOWER
I20260926 13:11:33.515014 14751 consensus_queue.cc:260] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } }
I20260926 13:11:33.515301 14751 ts_tablet_manager.cc:1428] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260926 13:11:33.518618 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.507406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:53485)
0926 13:11:33.507677 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.507716 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.508167 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.508737 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.508741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.509020 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:33.509209 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.509214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.510722 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.510723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.511645 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.511648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.512486 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.512487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.512606 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.513152 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:33.513161 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:33.514892 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:33.516920 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:33.516928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:33.516930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:33.517195 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:33.517513 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:33.517514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:33.517514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:33.517633 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:33.518103 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:33.518109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:33.518225 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.518310 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.518476 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":90,"threads_started":1}
I20260926 13:11:33.518733 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.507494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53485 (local address 127.0.98.62:33361)
0926 13:11:33.508230 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.508243 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.508301 (+ 58us) server_negotiation.cc:407] Connection header received
0926 13:11:33.508469 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.508476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.508543 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.508708 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:33.509341 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.510584 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.511744 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.512327 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.512474 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.515027 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:33.515032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:33.515032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:33.515045 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:33.516819 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:33.517290 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:33.517294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:33.517295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:33.517423 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:33.517703 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:33.517704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:33.517704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:33.518032 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:33.518120 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.518393 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.518542 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":65,"threads_started":1}
I20260926 13:11:33.519361 14859 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33361
I20260926 13:11:33.519455 14859 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:33.519634 14859 heartbeater.cc:511] Master 127.0.98.62:33361 requested a full tablet report, sending...
I20260926 13:11:33.520437 14671 ts_manager.cc:194] Registered new tserver with Master: a552129a0c924f4a8e0fc407905eb409 (127.0.98.1:46411)
I20260926 13:11:33.521395 14671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53485
I20260926 13:11:33.522223 14859 heartbeater.cc:503] Master 127.0.98.62:33361 was elected leader, sending a full tablet report...
Sep 26 13:11:33 dist-test-slave-2dw5 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:11:33.530181 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.521015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46411 (local address 127.0.0.1:47456)
0926 13:11:33.521240 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.521274 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.521338 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.521619 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.521624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.521950 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:33.522031 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.522035 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.523298 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.523298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.523955 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.523958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.524588 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.524592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.524657 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.525250 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:33.525255 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:33.526875 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:33.528995 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:33.529003 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:33.529005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:33.529224 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:33.529436 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:33.529437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:33.529437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:33.529501 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:33.529821 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:33.529828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:33.529883 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.529971 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.530086 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260926 13:11:33.530220 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.521115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.0.98.1:46411)
0926 13:11:33.521463 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.521468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.521477 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:11:33.521509 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.521511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.521525 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.522117 (+ 592us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:33.522174 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.523220 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.524041 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.524501 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.524573 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.526958 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:33.526968 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:33.526968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:33.526977 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:33.528893 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:33.529299 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:33.529302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:33.529303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:33.529366 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:33.529557 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:33.529557 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:33.529558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:33.529736 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:33.529815 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.530035 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.530122 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":82,"threads_started":1}
I20260926 13:11:33.530638 14793 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1ba8774a734282b181f7d0e946520e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47456
I20260926 13:11:33.531306 14868 tablet_replica.cc:330] stopping tablet replica
I20260926 13:11:33.531409 14868 raft_consensus.cc:2229] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:11:33.531486 14868 raft_consensus.cc:2258] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:11:33.531781 14868 ts_tablet_manager.cc:1905] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:11:33.533872 14868 ts_tablet_manager.cc:1918] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:11:33.533946 14868 log.cc:1196] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal/wals/4b1ba8774a734282b181f7d0e946520e
I20260926 13:11:33.534178 14868 ts_tablet_manager.cc:1939] T 4b1ba8774a734282b181f7d0e946520e P a552129a0c924f4a8e0fc407905eb409: Deleting consensus metadata
I20260926 13:11:33.534726 14657 catalog_manager.cc:4841] TS a552129a0c924f4a8e0fc407905eb409 (127.0.98.1:46411): tablet 4b1ba8774a734282b181f7d0e946520e (table test-table [id=922bb57466c84597948977fae9f6f2e0]) successfully deleted
W20260926 13:11:33.599103 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:33.599334 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:33.599370 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:33.600639 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:33.600689 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:33.600726 14864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:11:33.602483 14864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33679
--tserver_master_addrs=127.0.98.62:33361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:33.602744 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:33.603008 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:33.608306 14874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:33.608353 14876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:33.608394 14873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:33.608656 14864 server_base.cc:1034] running on GCE node
I20260926 13:11:33.608922 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:33.609229 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:33.610373 14864 hybrid_clock.cc:648] HybridClock initialized: now 1790428293610356 us; error 31 us; skew 500 ppm
Sep 26 13:11:33 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:33.613097 14864 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:11:33.613566 14864 webserver.cc:460] Webserver started at http://127.0.98.2:33679/ using document root <none> and password file <none>
I20260926 13:11:33.613767 14864 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:33.613817 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:33.615209 14864 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 13:11:33.615869 14883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:33.616153 14864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:33.616241 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "9e733b9d5c1b4615ae610b08f0995ba3"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "be537616b429c6b6e205b19ca3666d5a"
server_key_iv: "97815b5bbd01188f514eeb490f55666e"
server_key_version: "encryptionkey@0"
I20260926 13:11:33.616603 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:33.624495 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:33.627200 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:33.627355 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:33.627591 14864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:33.627877 14864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:33.627911 14864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:33.627948 14864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:33.627972 14864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:33.633973 14864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34727
I20260926 13:11:33.634023 14996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34727 every 8 connection(s)
I20260926 13:11:33.634378 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:11:33.639156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 14864
I20260926 13:11:33.639439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40303
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33361
--webserver_port=33713
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:11:33 dist-test-slave-2dw5 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:33.643959 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.635194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42655 (local address 127.0.98.62:33361)
0926 13:11:33.635251 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.635253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.635439 (+ 186us) server_negotiation.cc:407] Connection header received
0926 13:11:33.635920 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.635921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.635935 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.635981 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:33.636876 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.637413 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.638425 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.638628 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.638677 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.641113 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:33.641115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:33.641115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:33.641122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:33.642569 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:33.643041 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:33.643042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:33.643042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:33.643097 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:33.643320 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:33.643321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:33.643321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:33.643439 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:33.643495 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.643767 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.643863 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:11:33.644125 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.635190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:42655)
0926 13:11:33.635330 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.635368 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.635841 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.636036 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.636040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.636588 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:33.636796 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.636801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.637484 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.637485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.638343 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.638346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.638705 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.638706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.638814 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.639397 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:33.639406 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:33.640984 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:33.642686 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:33.642693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:33.642695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:33.642962 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:33.643157 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:33.643157 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:33.643158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:33.643262 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:33.643495 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:33.643502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:33.643648 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.643737 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.643899 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260926 13:11:33.644637 14997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33361
I20260926 13:11:33.644728 14997 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:33.644914 14997 heartbeater.cc:511] Master 127.0.98.62:33361 requested a full tablet report, sending...
I20260926 13:11:33.645309 14671 ts_manager.cc:194] Registered new tserver with Master: 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727)
I20260926 13:11:33.645812 14671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42655
W20260926 13:11:33.722224 15000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:33.722435 15000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:33.722458 15000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:33.723681 15000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:33.723728 15000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:33.723754 15000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:11:33.725283 15000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33713
--tserver_master_addrs=127.0.98.62:33361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:33.725512 15000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:33.725718 15000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:33.730886 15008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:33.730888 15006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:33.731199 15000 server_base.cc:1034] running on GCE node
W20260926 13:11:33.730886 15005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:33.731436 15000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:33.731688 15000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:33.732872 15000 hybrid_clock.cc:648] HybridClock initialized: now 1790428293732852 us; error 29 us; skew 500 ppm
Sep 26 13:11:33 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:33.735502 15000 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:11:33.736027 15000 webserver.cc:460] Webserver started at http://127.0.98.3:33713/ using document root <none> and password file <none>
I20260926 13:11:33.736225 15000 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:33.736286 15000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:33.737663 15000 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:33.738315 15015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:33.738574 15000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:33.738708 15000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "e96afd1c2edb456782394997a593285c"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "fff86af2c701de71752312b78415e21f"
server_key_iv: "3fbf4e1eaf2a0a1b8994222fea08c9fb"
server_key_version: "encryptionkey@0"
I20260926 13:11:33.739114 15000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:33.757527 15000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:33.759481 15000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:33.759675 15000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:33.759910 15000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:33.760211 15000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:33.760244 15000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:33.760273 15000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:33.760296 15000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:33.765937 15000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40303
I20260926 13:11:33.766053 15128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40303 every 8 connection(s)
I20260926 13:11:33.766340 15000 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:11:33 dist-test-slave-2dw5 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:33.775022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 15000
I20260926 13:11:33.775759 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.767090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52791 (local address 127.0.98.62:33361)
0926 13:11:33.767137 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.767139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.767365 (+ 226us) server_negotiation.cc:407] Connection header received
0926 13:11:33.767875 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.767876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.767886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.767926 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:33.768806 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.769375 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.770424 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.770685 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.770741 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.772979 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:33.772980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:33.772981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:33.772988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:33.774415 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:33.774782 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:33.774783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:33.774783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:33.774832 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:33.775050 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:33.775051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:33.775052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:33.775185 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:33.775249 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.775589 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.775674 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:11:33.775991 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.767102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:52791)
0926 13:11:33.767241 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.767277 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.767783 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.767989 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.767992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.768553 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:33.768742 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.768747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.769509 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.769510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.770336 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.770338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.770754 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.770755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.770894 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.771408 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:33.771417 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:33.772894 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:33.774501 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:33.774508 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:33.774511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:33.774723 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:33.774894 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:33.774894 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:33.774895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:33.774996 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:33.775261 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:33.775269 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:33.775452 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.775561 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.775746 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260926 13:11:33.776489 15129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33361
I20260926 13:11:33.776595 15129 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:33.776782 15129 heartbeater.cc:511] Master 127.0.98.62:33361 requested a full tablet report, sending...
I20260926 13:11:33.777164 14671 ts_manager.cc:194] Registered new tserver with Master: e96afd1c2edb456782394997a593285c (127.0.98.3:40303)
I20260926 13:11:33.777715 14671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52791
I20260926 13:11:33.781725 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.775406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36442 (local address 127.0.98.62:33361)
0926 13:11:33.775514 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.775516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.775627 (+ 111us) server_negotiation.cc:407] Connection header received
0926 13:11:33.775678 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.775680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.775693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.775756 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:33.776536 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.777227 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.778153 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.778438 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.778502 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.779745 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:33.779747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:33.779748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:33.779754 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:33.780829 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:33.781182 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:33.781183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:33.781183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:33.781229 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:33.781351 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:33.781352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:33.781352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:33.781437 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:33.781541 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.781567 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.781660 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
I20260926 13:11:33.782375 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:11:33.784920 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.782703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.0.98.62:33361)
0926 13:11:33.782751 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.782753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.782797 (+ 44us) server_negotiation.cc:407] Connection header received
0926 13:11:33.782890 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.782891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.782927 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.782981 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:11:33.783081 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.783554 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.784251 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.784458 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.784510 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.784651 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:11:33.784748 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:11:33.784817 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.784834 (+ 17us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.784856 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:11:33.785200 14671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:11:33.786823 14671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:11:33.789095 14793 tablet_service.cc:1467] Processing CreateTablet for tablet 0e037873d1ab4bca81b212869e079b12 (DEFAULT_TABLE table=test-table [id=b66e8374ef2f4d258580983a0329a4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:11:33.789299 14793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e037873d1ab4bca81b212869e079b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:33.790789 14751 tablet_bootstrap.cc:492] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409: Bootstrap starting.
I20260926 13:11:33.791129 14751 tablet_bootstrap.cc:654] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:33.791661 14751 tablet_bootstrap.cc:492] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409: No bootstrap required, opened a new log
I20260926 13:11:33.791718 14751 ts_tablet_manager.cc:1397] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:33.792038 14751 raft_consensus.cc:348] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:33.792114 14751 raft_consensus.cc:374] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:33.792165 14751 raft_consensus.cc:729] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552129a0c924f4a8e0fc407905eb409, State: Initialized, Role: FOLLOWER
I20260926 13:11:33.792207 14751 consensus_queue.cc:260] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:33.792290 14751 ts_tablet_manager.cc:1428] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:33.792696 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.788777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34727 (local address 127.0.0.1:60256)
0926 13:11:33.788819 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.788830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.788864 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.789247 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.789248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.789249 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:33.789296 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.789297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.790525 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.790526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.791526 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.791529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.792442 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.792443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.792514 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.792537 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.792588 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.792637 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 13:11:33.793107 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.788901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.0.98.2:34727)
0926 13:11:33.789058 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.789063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.789073 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:11:33.789112 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.789114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.789139 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.789367 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:33.789442 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.790415 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.791601 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.792360 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.792413 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.792477 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.792964 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.793012 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260926 13:11:33.793731 14931 tablet_service.cc:1467] Processing CreateTablet for tablet 0e037873d1ab4bca81b212869e079b12 (DEFAULT_TABLE table=test-table [id=b66e8374ef2f4d258580983a0329a4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:11:33.794021 14931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e037873d1ab4bca81b212869e079b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:33.794358 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.788827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40303 (local address 127.0.0.1:58782)
0926 13:11:33.790886 (+ 2059us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.790896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.790944 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.791121 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.791122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.791132 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:33.791169 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.791171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.792394 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.792395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.793416 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.793420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.794123 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.794124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.794187 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.794191 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.794252 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.794291 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2041,"thread_start_us":110,"threads_started":1}
I20260926 13:11:33.794390 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.789046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.0.98.3:40303)
0926 13:11:33.789228 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.789232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.791000 (+ 1768us) server_negotiation.cc:407] Connection header received
0926 13:11:33.791039 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.791041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.791053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.791113 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:33.791227 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.792256 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.793509 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.794041 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.794109 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.794166 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.794267 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.794308 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260926 13:11:33.795073 15063 tablet_service.cc:1467] Processing CreateTablet for tablet 0e037873d1ab4bca81b212869e079b12 (DEFAULT_TABLE table=test-table [id=b66e8374ef2f4d258580983a0329a4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:11:33.795408 15063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e037873d1ab4bca81b212869e079b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:33.797526 15139 tablet_bootstrap.cc:492] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3: Bootstrap starting.
I20260926 13:11:33.798220 15139 tablet_bootstrap.cc:654] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:33.798533 15140 tablet_bootstrap.cc:492] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c: Bootstrap starting.
I20260926 13:11:33.798748 15139 log.cc:824] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:33.799252 15140 tablet_bootstrap.cc:654] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:33.799635 15139 tablet_bootstrap.cc:492] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3: No bootstrap required, opened a new log
I20260926 13:11:33.799723 15139 ts_tablet_manager.cc:1397] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:11:33.799811 15140 log.cc:824] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:33.800562 15140 tablet_bootstrap.cc:492] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c: No bootstrap required, opened a new log
I20260926 13:11:33.800631 15140 ts_tablet_manager.cc:1397] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:11:33.801609 15139 raft_consensus.cc:348] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:33.801796 15139 raft_consensus.cc:374] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:33.801841 15139 raft_consensus.cc:729] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e733b9d5c1b4615ae610b08f0995ba3, State: Initialized, Role: FOLLOWER
I20260926 13:11:33.801970 15139 consensus_queue.cc:260] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:33.802243 15139 ts_tablet_manager.cc:1428] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 13:11:33.802263 14997 heartbeater.cc:503] Master 127.0.98.62:33361 was elected leader, sending a full tablet report...
I20260926 13:11:33.802531 15140 raft_consensus.cc:348] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:33.802682 15140 raft_consensus.cc:374] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:33.802747 15140 raft_consensus.cc:729] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e96afd1c2edb456782394997a593285c, State: Initialized, Role: FOLLOWER
I20260926 13:11:33.802897 15140 consensus_queue.cc:260] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:33.803171 15140 ts_tablet_manager.cc:1428] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:11:33.803249 15129 heartbeater.cc:503] Master 127.0.98.62:33361 was elected leader, sending a full tablet report...
W20260926 13:11:33.884975 14998 tablet.cc:2369] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:11:33.899518 15144 raft_consensus.cc:482] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:11:33.899676 15144 raft_consensus.cc:504] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:33.900090 15144 leader_election.cc:288] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727), a552129a0c924f4a8e0fc407905eb409 (127.0.98.1:46411)
I20260926 13:11:33.903326 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.900388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37759 (local address 127.0.98.2:34727)
0926 13:11:33.900440 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.900442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.900453 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:11:33.900485 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.900485 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.900498 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.900612 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:33.900753 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.901329 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.902378 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.902961 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.903013 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.903022 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.903230 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.903271 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 13:11:33.903318 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.900288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34727 (local address 127.0.98.3:37759)
0926 13:11:33.900348 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.900360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.900407 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.900617 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.900618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.900620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:33.900679 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.900681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.901427 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.901429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.902275 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.902277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.903043 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.903044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.903124 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.903129 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.903189 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.903233 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 13:11:33.903718 14951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e037873d1ab4bca81b212869e079b12" candidate_uuid: "e96afd1c2edb456782394997a593285c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" is_pre_election: true
I20260926 13:11:33.903865 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.900389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46411 (local address 127.0.98.3:37641)
0926 13:11:33.900622 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.900633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:33.900691 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:33.900837 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:33.900837 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:33.900839 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:33.900906 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.900907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.901753 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.901754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.902672 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:33.902675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.903625 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.903626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:33.903708 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.903712 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:11:33.903763 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 13:11:33.903794 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":163,"threads_started":1}
I20260926 13:11:33.903893 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.900480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37641 (local address 127.0.98.1:46411)
0926 13:11:33.900553 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.900555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.900699 (+ 144us) server_negotiation.cc:407] Connection header received
0926 13:11:33.900730 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.900731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.900762 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.900819 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:33.901064 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.901646 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.902768 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.903485 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.903544 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.903652 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.903785 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.903835 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:11:33.903899 14951 raft_consensus.cc:2454] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e96afd1c2edb456782394997a593285c in term 0.
I20260926 13:11:33.904131 15018 leader_election.cc:302] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e733b9d5c1b4615ae610b08f0995ba3, e96afd1c2edb456782394997a593285c; no voters:
I20260926 13:11:33.904258 15144 raft_consensus.cc:2781] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:11:33.904320 15144 raft_consensus.cc:482] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:11:33.904340 15144 raft_consensus.cc:3037] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:33.904436 14813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e037873d1ab4bca81b212869e079b12" candidate_uuid: "e96afd1c2edb456782394997a593285c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552129a0c924f4a8e0fc407905eb409" is_pre_election: true
I20260926 13:11:33.904644 14813 raft_consensus.cc:2454] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e96afd1c2edb456782394997a593285c in term 0.
I20260926 13:11:33.905192 15144 raft_consensus.cc:504] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:33.905304 15144 leader_election.cc:288] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [CANDIDATE]: Term 1 election: Requested vote from peers 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727), a552129a0c924f4a8e0fc407905eb409 (127.0.98.1:46411)
I20260926 13:11:33.905501 14813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e037873d1ab4bca81b212869e079b12" candidate_uuid: "e96afd1c2edb456782394997a593285c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a552129a0c924f4a8e0fc407905eb409"
I20260926 13:11:33.905517 14951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e037873d1ab4bca81b212869e079b12" candidate_uuid: "e96afd1c2edb456782394997a593285c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e733b9d5c1b4615ae610b08f0995ba3"
I20260926 13:11:33.905570 14813 raft_consensus.cc:3037] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:33.905604 14951 raft_consensus.cc:3037] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:33.906222 14813 raft_consensus.cc:2454] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e96afd1c2edb456782394997a593285c in term 1.
I20260926 13:11:33.906333 14951 raft_consensus.cc:2454] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e96afd1c2edb456782394997a593285c in term 1.
I20260926 13:11:33.906373 15018 leader_election.cc:302] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a552129a0c924f4a8e0fc407905eb409, e96afd1c2edb456782394997a593285c; no voters:
I20260926 13:11:33.906466 15144 raft_consensus.cc:2781] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:11:33.906620 15144 raft_consensus.cc:686] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 1 LEADER]: Becoming Leader. State: Replica: e96afd1c2edb456782394997a593285c, State: Running, Role: LEADER
I20260926 13:11:33.906697 15144 consensus_queue.cc:237] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:33.907637 14671 catalog_manager.cc:5495] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c reported cstate change: term changed from 0 to 1, leader changed from <none> to e96afd1c2edb456782394997a593285c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e96afd1c2edb456782394997a593285c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } health_report { overall_health: UNKNOWN } } }
I20260926 13:11:33.915822 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.913332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.0.98.3:40303)
0926 13:11:33.913386 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:33.913388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:33.913395 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:11:33.913458 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:33.913459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:33.913478 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:33.913531 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:11:33.913646 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.914098 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.914887 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:33.915196 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:33.915266 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:33.915436 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:11:33.915585 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:11:33.915650 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:33.915721 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 13:11:33.915741 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:11:33.917796 14951 raft_consensus.cc:1261] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3 [term 1 FOLLOWER]: Refusing update from remote peer e96afd1c2edb456782394997a593285c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:11:33.917797 14813 raft_consensus.cc:1261] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Refusing update from remote peer e96afd1c2edb456782394997a593285c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:11:33.918184 15146 consensus_queue.cc:1035] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:11:33.918332 15144 consensus_queue.cc:1035] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:11:33.920547 15149 mvcc.cc:205] Tried to move back new op lower bound from 7333594291884359680 to 7333594291842568192. Current Snapshot: MvccSnapshot[applied={T|T < 7333594291884359680}]
I20260926 13:11:33.921700 15150 mvcc.cc:205] Tried to move back new op lower bound from 7333594291884359680 to 7333594291842568192. Current Snapshot: MvccSnapshot[applied={T|T < 7333594291884359680 or (T in {7333594291884359680})}]
I20260926 13:11:33.923626 14670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:11:33.923725 14670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:11:33.924443 14670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Sending DeleteTablet for 3 replicas of tablet 0e037873d1ab4bca81b212869e079b12
I20260926 13:11:33.924749 15063 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e037873d1ab4bca81b212869e079b12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58782
I20260926 13:11:33.924867 14793 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e037873d1ab4bca81b212869e079b12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47456
I20260926 13:11:33.924914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 14709
I20260926 13:11:33.924978 14868 tablet_replica.cc:330] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409: stopping tablet replica
I20260926 13:11:33.925122 14868 raft_consensus.cc:2229] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:11:33.925184 14868 raft_consensus.cc:2258] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:11:33.925850 14931 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e037873d1ab4bca81b212869e079b12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60256
I20260926 13:11:33.925997 15158 tablet_replica.cc:330] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c: stopping tablet replica
I20260926 13:11:33.926093 15158 raft_consensus.cc:2229] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:11:33.926199 15159 tablet_replica.cc:330] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3: stopping tablet replica
I20260926 13:11:33.926237 15158 raft_consensus.cc:2258] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:11:33.926554 15159 raft_consensus.cc:2229] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:11:33.926668 15158 ts_tablet_manager.cc:1905] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:11:33.926754 15159 raft_consensus.cc:2258] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:11:33.927315 15159 ts_tablet_manager.cc:1905] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:11:33.929092 15159 ts_tablet_manager.cc:1918] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:11:33.929155 15158 ts_tablet_manager.cc:1918] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:11:33.929276 15158 log.cc:1196] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/0e037873d1ab4bca81b212869e079b12
I20260926 13:11:33.929284 15159 log.cc:1196] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/0e037873d1ab4bca81b212869e079b12
I20260926 13:11:33.929591 15158 ts_tablet_manager.cc:1939] T 0e037873d1ab4bca81b212869e079b12 P e96afd1c2edb456782394997a593285c: Deleting consensus metadata
I20260926 13:11:33.929767 15159 ts_tablet_manager.cc:1939] T 0e037873d1ab4bca81b212869e079b12 P 9e733b9d5c1b4615ae610b08f0995ba3: Deleting consensus metadata
I20260926 13:11:33.930147 14656 catalog_manager.cc:4841] TS e96afd1c2edb456782394997a593285c (127.0.98.3:40303): tablet 0e037873d1ab4bca81b212869e079b12 (table test-table [id=b66e8374ef2f4d258580983a0329a4f2]) successfully deleted
I20260926 13:11:33.930325 14657 catalog_manager.cc:4841] TS 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727): tablet 0e037873d1ab4bca81b212869e079b12 (table test-table [id=b66e8374ef2f4d258580983a0329a4f2]) successfully deleted
W20260926 13:11:33.932368 14657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46411 (error 108)
I20260926 13:11:33.932529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 14864
I20260926 13:11:33.933923 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:33.933291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46411 (local address 127.0.0.1:47470)
0926 13:11:33.933850 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:33.933891 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533}
W20260926 13:11:33.934067 14657 catalog_manager.cc:4584] TS a552129a0c924f4a8e0fc407905eb409 (127.0.98.1:46411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e037873d1ab4bca81b212869e079b12: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46411: connect: Connection refused (error 111)
I20260926 13:11:33.937935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 15000
I20260926 13:11:33.942241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 14638
I20260926 13:11:33.946969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33361
--webserver_interface=127.0.98.62
--webserver_port=42521
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:34.029470 15161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:34.029713 15161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:34.029739 15161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:34.031008 15161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:11:34.031054 15161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:34.031066 15161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:34.031077 15161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:11:34.031087 15161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:11:34.032706 15161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:34.032944 15161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:34.033142 15161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:34.038028 15166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:34.038148 15169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:34.038038 15167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:34.038260 15161 server_base.cc:1034] running on GCE node
I20260926 13:11:34.038506 15161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:34.038792 15161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:34.039978 15161 hybrid_clock.cc:648] HybridClock initialized: now 1790428294039943 us; error 47 us; skew 500 ppm
Sep 26 13:11:34 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:34.042976 15161 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:11:34.043442 15161 webserver.cc:460] Webserver started at http://127.0.98.62:42521/ using document root <none> and password file <none>
I20260926 13:11:34.043641 15161 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:34.043684 15161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:34.044931 15161 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:34.045648 15176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:34.045787 15161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:11:34.045866 15161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "800ea0684eca4979b8005059382fdf42"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "1c9f21c92e4ad83b92a66c496ed356ff"
server_key_iv: "f31f5779eff6f5d084d3da46044b0578"
server_key_version: "encryptionkey@0"
I20260926 13:11:34.046272 15161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:34.052664 15161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:34.054647 15161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:34.054760 15161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:34.058425 15161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33361
I20260926 13:11:34.058468 15228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33361 every 8 connection(s)
I20260926 13:11:34.058807 15161 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:11:34.060472 15229 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:11:34.061088 15229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Bootstrap starting.
I20260926 13:11:34.061831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 15161
I20260926 13:11:34.062115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46411
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33361
--webserver_port=45863
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:11:34.069537 15229 log.cc:824] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:34.071745 15229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:11:34.072006 15229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42: Bootstrap complete.
I20260926 13:11:34.074026 15229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260926 13:11:34.074247 15229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800ea0684eca4979b8005059382fdf42, State: Initialized, Role: FOLLOWER
I20260926 13:11:34.074383 15229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260926 13:11:34.074438 15229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:11:34.074484 15229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:11:34.074528 15229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 2 FOLLOWER]: Advancing to term 3
I20260926 13:11:34.075562 15229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260926 13:11:34.075656 15229 leader_election.cc:302] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 800ea0684eca4979b8005059382fdf42; no voters:
I20260926 13:11:34.075819 15229 leader_election.cc:288] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 13:11:34.075863 15233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 13:11:34.075971 15233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [term 3 LEADER]: Becoming Leader. State: Replica: 800ea0684eca4979b8005059382fdf42, State: Running, Role: LEADER
I20260926 13:11:34.076056 15233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } }
I20260926 13:11:34.076087 15229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:11:34.076330 15234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "800ea0684eca4979b8005059382fdf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } } }
I20260926 13:11:34.076413 15234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:34.076324 15233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 800ea0684eca4979b8005059382fdf42. Latest consensus state: current_term: 3 leader_uuid: "800ea0684eca4979b8005059382fdf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "800ea0684eca4979b8005059382fdf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33361 } } }
I20260926 13:11:34.076588 15233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 800ea0684eca4979b8005059382fdf42 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:34.076994 15238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:11:34.077737 15238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:11:34.077839 15238 catalog_manager.cc:1259] Loaded cluster ID: 5bcf8929a6ca4eca94008f2f84826c8e
I20260926 13:11:34.077934 15238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 13:11:34.078316 15238 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6cc43459ad google::LogMessage::Fail() at ??:0
@ 0x7f6cc434888a google::LogMessage::SendToLog() at ??:0
@ 0x7f6cc43454de google::LogMessage::Flush() at ??:0
@ 0x7f6cc4346a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6ccc2660f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6ccc26f8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6cc62130d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6cc6208954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6cc706c6db start_thread at ??:0
@ 0x7f6cc2e4361f clone at ??:0
W20260926 13:11:34.153607 15231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:34.153805 15231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:34.153826 15231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:34.155009 15231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:34.155048 15231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:34.155074 15231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:11:34.156656 15231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45863
--tserver_master_addrs=127.0.98.62:33361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:34.156886 15231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:34.157104 15231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:34.162313 15268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:34.162324 15266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:34.162325 15265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:34.162587 15231 server_base.cc:1034] running on GCE node
I20260926 13:11:34.162771 15231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:34.163087 15231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:34.164233 15231 hybrid_clock.cc:648] HybridClock initialized: now 1790428294164218 us; error 32 us; skew 500 ppm
Sep 26 13:11:34 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:34.166857 15231 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:11:34.167317 15231 webserver.cc:460] Webserver started at http://127.0.98.1:45863/ using document root <none> and password file <none>
I20260926 13:11:34.167548 15231 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:34.167598 15231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:34.168962 15231 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 13:11:34.169665 15275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:34.169821 15231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:34.169906 15231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "a552129a0c924f4a8e0fc407905eb409"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "a6964fe0cecbe31f4ef045b086de1c32"
server_key_iv: "e56ac9f699469a007ffcde15711be130"
server_key_version: "encryptionkey@0"
I20260926 13:11:34.170276 15231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:34.182426 15231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:34.184617 15231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:34.184753 15231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:34.184985 15231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:34.185403 15282 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:11:34.187376 15231 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:11:34.187425 15231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 13:11:34.187458 15231 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:11:34.188150 15231 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:11:34.188184 15231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 13:11:34.188265 15282 tablet_bootstrap.cc:492] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409: Bootstrap starting.
I20260926 13:11:34.194689 15231 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46411
I20260926 13:11:34.194823 15389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46411 every 8 connection(s)
I20260926 13:11:34.195142 15231 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:11:34.197292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 15231
I20260926 13:11:34.197552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34727
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33361
--webserver_port=33679
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:11:34.198917 15282 log.cc:824] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:34.205735 15282 tablet_bootstrap.cc:492] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:11:34.206043 15282 tablet_bootstrap.cc:492] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409: Bootstrap complete.
I20260926 13:11:34.207216 15282 ts_tablet_manager.cc:1397] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.010s
I20260926 13:11:34.208745 15282 raft_consensus.cc:348] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:34.209079 15282 raft_consensus.cc:729] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552129a0c924f4a8e0fc407905eb409, State: Initialized, Role: FOLLOWER
I20260926 13:11:34.209254 15282 consensus_queue.cc:260] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:34.209544 15282 ts_tablet_manager.cc:1428] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260926 13:11:34.273022 15392 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:11:34.195850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:34471)
0926 13:11:34.196026 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:34.196069 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:34.196529 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:34.272795 (+ 76266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
W20260926 13:11:34.273339 15390 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33361 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33361: Client connection negotiation failed: client connection to 127.0.98.62:33361: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:11:34.273679 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:34.273545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:40123)
0926 13:11:34.273597 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:34.273649 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 13:11:34.274013 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:34.273913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:56559)
0926 13:11:34.273954 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:34.273991 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 13:11:34.274233 15390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:11:34.278558 15393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:34.278769 15393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:34.278801 15393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:34.280078 15393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:34.280129 15393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:34.280166 15393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:11:34.281713 15393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33679
--tserver_master_addrs=127.0.98.62:33361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:34.281970 15393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:34.282183 15393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:34.287071 15400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:34.287074 15401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:34.287182 15403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:34.287559 15393 server_base.cc:1034] running on GCE node
I20260926 13:11:34.287724 15393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:34.287925 15393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:34.289142 15393 hybrid_clock.cc:648] HybridClock initialized: now 1790428294289122 us; error 30 us; skew 500 ppm
Sep 26 13:11:34 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:34.291842 15393 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:11:34.292291 15393 webserver.cc:460] Webserver started at http://127.0.98.2:33679/ using document root <none> and password file <none>
I20260926 13:11:34.292520 15393 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:34.292567 15393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:34.293901 15393 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:11:34.294538 15410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:34.294672 15393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:11:34.294756 15393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "9e733b9d5c1b4615ae610b08f0995ba3"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "be537616b429c6b6e205b19ca3666d5a"
server_key_iv: "97815b5bbd01188f514eeb490f55666e"
server_key_version: "encryptionkey@0"
I20260926 13:11:34.295197 15393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:34.304049 15393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:34.306077 15393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:34.306237 15393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:34.306453 15393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:34.306740 15393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:34.306773 15393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:34.306802 15393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:34.306826 15393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:34.312520 15393 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34727
I20260926 13:11:34.312589 15523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34727 every 8 connection(s)
I20260926 13:11:34.312906 15393 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:11:34.313921 15526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:34.313557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:57373)
0926 13:11:34.313711 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:34.313781 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260926 13:11:34.314405 15524 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33361 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33361: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
I20260926 13:11:34.314702 15526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:34.314601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:52125)
0926 13:11:34.314659 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:34.314684 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 13:11:34.314958 15526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:34.314907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:34255)
0926 13:11:34.314928 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:34.314946 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 13:11:34.315099 15524 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:11:34.322471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 15393
I20260926 13:11:34.322708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40303
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33361
--webserver_port=33713
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:34.404569 15527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:34.404804 15527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:34.404830 15527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:34.406020 15527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:34.406057 15527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:34.406082 15527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:11:34.407547 15527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33713
--tserver_master_addrs=127.0.98.62:33361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:34.407752 15527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:34.407935 15527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:34.412873 15533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:34.412899 15535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:34.412873 15532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:34.413352 15527 server_base.cc:1034] running on GCE node
I20260926 13:11:34.413538 15527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:34.413739 15527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:34.414929 15527 hybrid_clock.cc:648] HybridClock initialized: now 1790428294414910 us; error 25 us; skew 500 ppm
Sep 26 13:11:34 dist-test-slave-2dw5 krb5kdc[14089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:34.417443 15527 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:11:34.417858 15527 webserver.cc:460] Webserver started at http://127.0.98.3:33713/ using document root <none> and password file <none>
I20260926 13:11:34.418033 15527 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:34.418071 15527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:34.419402 15527 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:11:34.420089 15542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:34.420302 15527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:11:34.420392 15527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "e96afd1c2edb456782394997a593285c"
format_stamp: "Formatted at 2026-09-26 13:11:32 on dist-test-slave-2dw5"
server_key: "fff86af2c701de71752312b78415e21f"
server_key_iv: "3fbf4e1eaf2a0a1b8994222fea08c9fb"
server_key_version: "encryptionkey@0"
I20260926 13:11:34.420739 15527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:34.436169 15527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:34.438519 15527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:34.438669 15527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:34.438934 15527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:34.439350 15527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:34.439388 15527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:34.439416 15527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:34.439433 15527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:34.445132 15527 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40303
I20260926 13:11:34.445212 15655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40303 every 8 connection(s)
I20260926 13:11:34.445526 15527 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:11:34.447073 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:34.446593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:47313)
0926 13:11:34.446782 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:34.446911 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260926 13:11:34.447429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 15527
W20260926 13:11:34.447614 15656 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33361 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33361: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
I20260926 13:11:34.447973 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:34.447888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:56039)
0926 13:11:34.447921 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:34.447946 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 13:11:34.448292 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:34.448221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:41135)
0926 13:11:34.448256 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:34.448276 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 13:11:34.448402 15656 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:11:35.274957 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:35.274584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:56017)
0926 13:11:35.274776 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:35.274880 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20260926 13:11:35.315795 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:35.315489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:52039)
0926 13:11:35.315653 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:35.315712 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260926 13:11:35.449106 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:35.448749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:55249)
0926 13:11:35.448914 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:35.448990 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260926 13:11:35.454814 15663 raft_consensus.cc:482] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:11:35.454988 15663 raft_consensus.cc:504] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:35.455468 15663 leader_election.cc:288] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727), e96afd1c2edb456782394997a593285c (127.0.98.3:40303)
Sep 26 13:11:35 dist-test-slave-2dw5 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 26 13:11:35 dist-test-slave-2dw5 krb5kdc[14089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260926 13:11:35.466881 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:35.455782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43253 (local address 127.0.98.3:40303)
0926 13:11:35.456040 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:35.456057 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:35.456219 (+ 162us) server_negotiation.cc:407] Connection header received
0926 13:11:35.456807 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:35.456811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:35.456867 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:35.457091 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:35.457817 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:35.458765 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:35.459978 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:35.460444 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:35.460560 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:35.463539 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:35.463544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:35.463545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:35.463558 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:35.465193 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:35.465608 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:35.465611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:35.465611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:35.465709 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:35.465939 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:35.465940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:35.465940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:35.466202 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:35.466278 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:35.466593 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 13:11:35.466728 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260926 13:11:35.466871 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:35.455683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34727 (local address 127.0.98.1:38385)
0926 13:11:35.455937 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:35.455950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:35.456034 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:35.457029 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:35.457035 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:35.457470 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:35.457700 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:35.457710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:35.458988 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:35.458990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:35.459894 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:35.459897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:35.460568 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:35.460569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:35.460670 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:35.461349 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:35.461354 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:35.463030 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:35.465189 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:35.465197 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:35.465199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:35.465454 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:35.465726 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:35.465727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:35.465727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:35.465851 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:35.466274 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:35.466281 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:35.466413 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 13:11:35.466540 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 13:11:35.466723 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 13:11:35.466903 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:35.455921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38385 (local address 127.0.98.2:34727)
0926 13:11:35.456173 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:35.456188 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:35.456246 (+ 58us) server_negotiation.cc:407] Connection header received
0926 13:11:35.456772 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:35.456774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:35.456821 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:35.456968 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:35.457849 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:35.458810 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:35.459993 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:35.460489 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:35.460594 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:35.463172 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:35.463177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:35.463178 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:35.463299 (+ 121us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:35.465090 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:35.465537 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:35.465541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:35.465541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:35.465654 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:35.465916 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:35.465917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:35.465918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:35.466200 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:35.466276 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:35.466613 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 13:11:35.466750 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260926 13:11:35.466979 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:35.455704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40303 (local address 127.0.98.1:43253)
0926 13:11:35.456127 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:35.456140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:35.456206 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:35.457050 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:35.457052 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:35.457457 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:35.457698 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:35.457705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:35.458896 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:35.458898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:35.459891 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:35.459893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:35.460533 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:35.460534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:35.460670 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:35.461300 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:35.461309 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:35.463405 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:35.465307 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:35.465308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:35.465309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:35.465517 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:35.465782 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:35.465783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:35.465783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:35.465869 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:35.466275 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:35.466284 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:35.466415 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 13:11:35.466516 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 13:11:35.466814 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":165,"thread_start_us":94,"threads_started":1}
W20260926 13:11:35.467980 15278 leader_election.cc:341] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727): Not found: Tablet not found: 0e037873d1ab4bca81b212869e079b12
W20260926 13:11:35.468590 15277 leader_election.cc:341] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e96afd1c2edb456782394997a593285c (127.0.98.3:40303): Not found: Tablet not found: 0e037873d1ab4bca81b212869e079b12
I20260926 13:11:35.468659 15277 leader_election.cc:302] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552129a0c924f4a8e0fc407905eb409; no voters: 9e733b9d5c1b4615ae610b08f0995ba3, e96afd1c2edb456782394997a593285c
I20260926 13:11:35.468784 15663 raft_consensus.cc:2726] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:11:36.275914 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:36.275594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:59801)
0926 13:11:36.275778 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:36.275846 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":67,"threads_started":1}
I20260926 13:11:36.316655 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:36.316425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:41571)
0926 13:11:36.316572 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:36.316609 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260926 13:11:36.450150 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:36.449864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:50785)
0926 13:11:36.450038 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:36.450093 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260926 13:11:37.276701 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:37.276482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:46169)
0926 13:11:37.276610 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:37.276652 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260926 13:11:37.317430 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:37.317171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:36291)
0926 13:11:37.317337 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:37.317382 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260926 13:11:37.335013 15672 raft_consensus.cc:482] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:11:37.335108 15672 raft_consensus.cc:504] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:37.335287 15672 leader_election.cc:288] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727), e96afd1c2edb456782394997a593285c (127.0.98.3:40303)
W20260926 13:11:37.335660 15278 leader_election.cc:341] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727): Not found: Tablet not found: 0e037873d1ab4bca81b212869e079b12
W20260926 13:11:37.335796 15277 leader_election.cc:341] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e96afd1c2edb456782394997a593285c (127.0.98.3:40303): Not found: Tablet not found: 0e037873d1ab4bca81b212869e079b12
I20260926 13:11:37.335836 15277 leader_election.cc:302] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552129a0c924f4a8e0fc407905eb409; no voters: 9e733b9d5c1b4615ae610b08f0995ba3, e96afd1c2edb456782394997a593285c
I20260926 13:11:37.335903 15672 raft_consensus.cc:2726] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:11:37.450958 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:37.450688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:60665)
0926 13:11:37.450836 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:37.450912 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260926 13:11:38.277639 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:38.277338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:35449)
0926 13:11:38.277530 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:38.277590 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260926 13:11:38.318360 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:38.317949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:54475)
0926 13:11:38.318162 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:38.318236 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
I20260926 13:11:38.451850 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:38.451563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:42819)
0926 13:11:38.451740 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:38.451803 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260926 13:11:38.909216 15677 raft_consensus.cc:482] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:11:38.909356 15677 raft_consensus.cc:504] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:38.909572 15677 leader_election.cc:288] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727), e96afd1c2edb456782394997a593285c (127.0.98.3:40303)
W20260926 13:11:38.910138 15278 leader_election.cc:341] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727): Not found: Tablet not found: 0e037873d1ab4bca81b212869e079b12
W20260926 13:11:38.910286 15277 leader_election.cc:341] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e96afd1c2edb456782394997a593285c (127.0.98.3:40303): Not found: Tablet not found: 0e037873d1ab4bca81b212869e079b12
I20260926 13:11:38.910367 15277 leader_election.cc:302] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552129a0c924f4a8e0fc407905eb409; no voters: 9e733b9d5c1b4615ae610b08f0995ba3, e96afd1c2edb456782394997a593285c
I20260926 13:11:38.910629 15677 raft_consensus.cc:2726] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:11:39.278700 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:39.278379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:59267)
0926 13:11:39.278574 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:39.278637 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":80,"threads_started":1}
I20260926 13:11:39.319330 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:39.319019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:53781)
0926 13:11:39.319190 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:39.319271 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260926 13:11:39.452741 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:39.452493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:40397)
0926 13:11:39.452643 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:39.452679 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
I20260926 13:11:40.279487 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:40.279211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:49987)
0926 13:11:40.279382 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:40.279435 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260926 13:11:40.320088 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:40.319872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:51619)
0926 13:11:40.320000 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:40.320042 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260926 13:11:40.453501 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:40.453241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:51999)
0926 13:11:40.453410 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:40.453451 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260926 13:11:40.531014 15684 raft_consensus.cc:482] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:11:40.531123 15684 raft_consensus.cc:504] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:40.531317 15684 leader_election.cc:288] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727), e96afd1c2edb456782394997a593285c (127.0.98.3:40303)
W20260926 13:11:40.531735 15278 leader_election.cc:341] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727): Not found: Tablet not found: 0e037873d1ab4bca81b212869e079b12
W20260926 13:11:40.531813 15277 leader_election.cc:341] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e96afd1c2edb456782394997a593285c (127.0.98.3:40303): Not found: Tablet not found: 0e037873d1ab4bca81b212869e079b12
I20260926 13:11:40.531857 15277 leader_election.cc:302] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552129a0c924f4a8e0fc407905eb409; no voters: 9e733b9d5c1b4615ae610b08f0995ba3, e96afd1c2edb456782394997a593285c
I20260926 13:11:40.531934 15684 raft_consensus.cc:2726] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:11:41.280277 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:41.279985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:53893)
0926 13:11:41.280166 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:41.280223 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260926 13:11:41.321025 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:41.320601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:49265)
0926 13:11:41.320805 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:41.320869 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260926 13:11:41.454329 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:41.454022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:39557)
0926 13:11:41.454214 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:41.454274 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20260926 13:11:42.281231 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:42.280988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:58569)
0926 13:11:42.281144 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:42.281183 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260926 13:11:42.322050 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:42.321696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:44347)
0926 13:11:42.321921 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:42.322000 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":97,"threads_started":1}
I20260926 13:11:42.455343 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:42.454999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:43715)
0926 13:11:42.455234 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:42.455293 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":108,"threads_started":1}
I20260926 13:11:43.281885 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:43.281658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:49943)
0926 13:11:43.281805 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:43.281838 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260926 13:11:43.322932 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:43.322681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:55483)
0926 13:11:43.322813 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:43.322871 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260926 13:11:43.456080 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:43.455875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:43835)
0926 13:11:43.455990 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:43.456033 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260926 13:11:44.282785 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:44.282450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:32805)
0926 13:11:44.282635 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:44.282717 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260926 13:11:44.323900 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:44.323572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:57475)
0926 13:11:44.323795 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:44.323854 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":103,"threads_started":1}
I20260926 13:11:44.457038 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:44.456715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:57599)
0926 13:11:44.456894 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:44.456966 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
I20260926 13:11:45.283761 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:45.283439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:56251)
0926 13:11:45.283649 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:45.283711 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
I20260926 13:11:45.324689 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:45.324423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:41071)
0926 13:11:45.324585 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:45.324618 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260926 13:11:45.457942 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:45.457633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:34805)
0926 13:11:45.457801 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:45.457875 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260926 13:11:46.284652 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:46.284411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:46795)
0926 13:11:46.284542 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:46.284593 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260926 13:11:46.325585 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:46.325264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:50755)
0926 13:11:46.325452 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:46.325521 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260926 13:11:46.458966 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:46.458578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:49195)
0926 13:11:46.458791 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:46.458901 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260926 13:11:47.238256 15703 raft_consensus.cc:482] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:11:47.238390 15703 raft_consensus.cc:504] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e733b9d5c1b4615ae610b08f0995ba3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34727 } } peers { permanent_uuid: "e96afd1c2edb456782394997a593285c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40303 } } peers { permanent_uuid: "a552129a0c924f4a8e0fc407905eb409" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46411 } }
I20260926 13:11:47.238583 15703 leader_election.cc:288] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727), e96afd1c2edb456782394997a593285c (127.0.98.3:40303)
W20260926 13:11:47.239081 15278 leader_election.cc:341] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e733b9d5c1b4615ae610b08f0995ba3 (127.0.98.2:34727): Not found: Tablet not found: 0e037873d1ab4bca81b212869e079b12
W20260926 13:11:47.239174 15277 leader_election.cc:341] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e96afd1c2edb456782394997a593285c (127.0.98.3:40303): Not found: Tablet not found: 0e037873d1ab4bca81b212869e079b12
I20260926 13:11:47.239295 15277 leader_election.cc:302] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a552129a0c924f4a8e0fc407905eb409; no voters: 9e733b9d5c1b4615ae610b08f0995ba3, e96afd1c2edb456782394997a593285c
I20260926 13:11:47.239384 15703 raft_consensus.cc:2726] T 0e037873d1ab4bca81b212869e079b12 P a552129a0c924f4a8e0fc407905eb409 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:11:47.285678 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:47.285274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:57133)
0926 13:11:47.285549 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:47.285620 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":97,"threads_started":1}
I20260926 13:11:47.326633 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:47.326216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:49063)
0926 13:11:47.326422 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:47.326487 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
I20260926 13:11:47.459988 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:47.459765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:40115)
0926 13:11:47.459896 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:47.459939 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260926 13:11:48.286537 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:48.286289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:45567)
0926 13:11:48.286446 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:48.286492 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
I20260926 13:11:48.327364 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:48.327102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:57611)
0926 13:11:48.327274 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:48.327319 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
I20260926 13:11:48.460708 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:48.460492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:33829)
0926 13:11:48.460626 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:48.460660 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260926 13:11:49.287343 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:49.287137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.1:34485)
0926 13:11:49.287255 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:49.287294 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260926 13:11:49.328132 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:49.327892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.2:43879)
0926 13:11:49.328051 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:49.328086 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260926 13:11:49.453302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 15231
I20260926 13:11:49.457746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 15393
I20260926 13:11:49.461503 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:49.461270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33361 (local address 127.0.98.3:47191)
0926 13:11:49.461405 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:49.461444 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260926 13:11:49.462087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 15527
2026-09-26T13:11:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18229 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:11:49 dist-test-slave-2dw5 krb5kdc[15715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:11:49 dist-test-slave-2dw5 krb5kdc[15715](info): set up 2 sockets
Sep 26 13:11:49 dist-test-slave-2dw5 krb5kdc[15715](info): commencing operation
krb5kdc: starting...
W20260926 13:11:50.422924 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:11:50 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:11:50.542642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40789
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:50.623629 15728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:50.623852 15728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:50.623883 15728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:50.625063 15728 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 13:11:50.625110 15728 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 13:11:50.625125 15728 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 13:11:50.625140 15728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:11:50.625156 15728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:50.625174 15728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:50.625193 15728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:11:50.625216 15728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:11:50.626910 15728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:50.627182 15728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:50.627419 15728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:50.632848 15734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:50.632903 15728 server_base.cc:1034] running on GCE node
W20260926 13:11:50.632918 15733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:50.633129 15736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:50.633435 15728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:50.633716 15728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:50.634876 15728 hybrid_clock.cc:648] HybridClock initialized: now 1790428310634855 us; error 32 us; skew 500 ppm
Sep 26 13:11:50 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:50.637435 15728 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:11:50.637916 15728 webserver.cc:460] Webserver started at http://127.0.98.62:42261/ using document root <none> and password file <none>
I20260926 13:11:50.638211 15728 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:50.638262 15728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:50.638382 15728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:11:50.639500 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "3344c043df7e46ef9bbfe07db4a7c943"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "d6b61fb64de8d84cde4cf7ceb89b9bbc"
server_key_iv: "05897514493cd8878fd3b6e324ddc5d6"
server_key_version: "encryptionkey@0"
I20260926 13:11:50.639810 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "3344c043df7e46ef9bbfe07db4a7c943"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "d6b61fb64de8d84cde4cf7ceb89b9bbc"
server_key_iv: "05897514493cd8878fd3b6e324ddc5d6"
server_key_version: "encryptionkey@0"
I20260926 13:11:50.641139 15728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:11:50.641898 15743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:50.642118 15728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:50.642204 15728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "3344c043df7e46ef9bbfe07db4a7c943"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "d6b61fb64de8d84cde4cf7ceb89b9bbc"
server_key_iv: "05897514493cd8878fd3b6e324ddc5d6"
server_key_version: "encryptionkey@0"
I20260926 13:11:50.642267 15728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:50.651885 15728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:50.654037 15728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:50.654178 15728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:50.658070 15728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40789
I20260926 13:11:50.658108 15795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40789 every 8 connection(s)
I20260926 13:11:50.658429 15728 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:11:50.659279 15796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:50.661641 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Bootstrap starting.
I20260926 13:11:50.662201 15796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:50.662446 15796 log.cc:824] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:50.663192 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: No bootstrap required, opened a new log
I20260926 13:11:50.664773 15796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260926 13:11:50.664915 15796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:50.664937 15796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3344c043df7e46ef9bbfe07db4a7c943, State: Initialized, Role: FOLLOWER
I20260926 13:11:50.665050 15796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260926 13:11:50.665107 15796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:11:50.665139 15796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:11:50.665179 15796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:50.665706 15796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260926 13:11:50.665807 15796 leader_election.cc:302] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3344c043df7e46ef9bbfe07db4a7c943; no voters:
I20260926 13:11:50.665968 15796 leader_election.cc:288] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:11:50.666109 15799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:11:50.666222 15796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:11:50.666252 15799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 1 LEADER]: Becoming Leader. State: Replica: 3344c043df7e46ef9bbfe07db4a7c943, State: Running, Role: LEADER
I20260926 13:11:50.666369 15799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260926 13:11:50.666742 15800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3344c043df7e46ef9bbfe07db4a7c943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } } }
I20260926 13:11:50.666720 15801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3344c043df7e46ef9bbfe07db4a7c943. Latest consensus state: current_term: 1 leader_uuid: "3344c043df7e46ef9bbfe07db4a7c943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } } }
I20260926 13:11:50.666853 15800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:50.666922 15801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: This master's current role is: LEADER
I20260926 13:11:50.667285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 15728
I20260926 13:11:50.667385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:11:50.667496 15808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:11:50.667585 392 external_mini_cluster.cc:1428] Setting key fc9c359c67c2f266f466dde492b1b196
I20260926 13:11:50.668020 15808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:11:50.669724 15808 catalog_manager.cc:1347] Generated new cluster ID: be9185f14b4040dbae12cda1d987e95a
I20260926 13:11:50.669776 15808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:11:50 dist-test-slave-2dw5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:50.679685 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:50.668227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.0.98.62:40789)
0926 13:11:50.668462 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:50.668471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:50.668511 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:11:50.668647 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:50.668654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:50.668711 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:50.668844 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:50.669587 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:50.671124 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:50.672175 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:50.672871 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:50.673004 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:50.675768 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:50.675773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:50.675773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:50.675787 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:50.677741 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:50.678203 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:50.678206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:50.678206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:50.678319 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:50.678509 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:50.678509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:50.678509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:50.678819 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:50.679026 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:50.679160 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 13:11:50.679427 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":68,"threads_started":1}
I20260926 13:11:50.685089 15808 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:11:50.685820 15808 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:11:50.693709 15808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Generated new TSK 0
I20260926 13:11:50.693925 15808 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:11:50.722625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40789
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:50.804764 15822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:50.804972 15822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:50.804996 15822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:50.806293 15822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:50.806340 15822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:50.806367 15822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:11:50.807930 15822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:50.808171 15822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:50.808403 15822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:50.813565 15828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:50.813567 15827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:50.813625 15822 server_base.cc:1034] running on GCE node
W20260926 13:11:50.813624 15830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:50.814024 15822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:50.814317 15822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:50.815482 15822 hybrid_clock.cc:648] HybridClock initialized: now 1790428310815462 us; error 62 us; skew 500 ppm
Sep 26 13:11:50 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:50.818233 15822 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:11:50.818723 15822 webserver.cc:460] Webserver started at http://127.0.98.1:42721/ using document root <none> and password file <none>
I20260926 13:11:50.818984 15822 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:50.819051 15822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:50.819177 15822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:11:50.820221 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "c423f60803b4469a93d739027ebe5450"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "701ab8d8eff39f333cd700d7ef95e70e"
server_key_iv: "cf77ab6ea847bec8790a3388b2dad95f"
server_key_version: "encryptionkey@0"
I20260926 13:11:50.820549 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c423f60803b4469a93d739027ebe5450"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "701ab8d8eff39f333cd700d7ef95e70e"
server_key_iv: "cf77ab6ea847bec8790a3388b2dad95f"
server_key_version: "encryptionkey@0"
I20260926 13:11:50.821861 15822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:11:50.822772 15837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:50.823123 15822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:50.823213 15822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "c423f60803b4469a93d739027ebe5450"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "701ab8d8eff39f333cd700d7ef95e70e"
server_key_iv: "cf77ab6ea847bec8790a3388b2dad95f"
server_key_version: "encryptionkey@0"
I20260926 13:11:50.823282 15822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:50.835204 15822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:50.837220 15822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:50.837358 15822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:50.837581 15822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:50.837878 15822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:50.837918 15822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:50.837950 15822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:50.837973 15822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:50.843780 15822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45747
I20260926 13:11:50.843827 15950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45747 every 8 connection(s)
I20260926 13:11:50.844156 15822 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:11:50.847872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 15822
I20260926 13:11:50.848003 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:11:50.848284 392 external_mini_cluster.cc:1428] Setting key 5a3092f2c5d9b51916fd2afdc5bfcd24
Sep 26 13:11:50 dist-test-slave-2dw5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:50.854493 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:50.844838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45583 (local address 127.0.98.62:40789)
0926 13:11:50.844883 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:50.844885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:50.845160 (+ 275us) server_negotiation.cc:407] Connection header received
0926 13:11:50.845693 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:50.845694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:50.845710 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:50.845793 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:50.847070 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:50.847645 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:50.848959 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:50.849228 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:50.849288 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:50.851747 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:50.851749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:50.851750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:50.851758 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:50.853169 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:50.853559 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:50.853560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:50.853561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:50.853614 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:50.853841 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:50.853842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:50.853842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:50.853958 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:50.854012 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:50.854308 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 13:11:50.854400 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:11:50.854712 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:50.844795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:45583)
0926 13:11:50.844957 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:50.845069 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:50.845564 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:50.845886 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:50.845890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:50.846605 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:50.846819 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:50.846824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:50.847771 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:50.847772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:50.848889 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:50.848893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:50.849316 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:50.849316 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:50.849478 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:50.850145 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:50.850156 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:50.851645 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:50.853248 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:50.853257 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:50.853260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:50.853490 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:50.853669 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:50.853670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:50.853671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:50.853784 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:50.854012 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:50.854019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:50.854181 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 13:11:50.854278 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 13:11:50.854443 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260926 13:11:50.855167 15951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40789
I20260926 13:11:50.855259 15951 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:50.855441 15951 heartbeater.cc:511] Master 127.0.98.62:40789 requested a full tablet report, sending...
I20260926 13:11:50.855929 15760 ts_manager.cc:194] Registered new tserver with Master: c423f60803b4469a93d739027ebe5450 (127.0.98.1:45747)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:11:50.856562 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:11:50.874631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40789
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:50.956529 15956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:50.956776 15956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:50.956810 15956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:50.958082 15956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:50.958133 15956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:50.958168 15956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:11:50.959694 15956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:50.959926 15956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:50.960129 15956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:50.965523 15964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:50.965613 15962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:50.965620 15961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:50.965794 15956 server_base.cc:1034] running on GCE node
I20260926 13:11:50.966053 15956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:50.966292 15956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:50.967450 15956 hybrid_clock.cc:648] HybridClock initialized: now 1790428310967408 us; error 58 us; skew 500 ppm
Sep 26 13:11:50 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:50.970191 15956 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:11:50.970678 15956 webserver.cc:460] Webserver started at http://127.0.98.2:43221/ using document root <none> and password file <none>
I20260926 13:11:50.970939 15956 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:50.970994 15956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:50.971132 15956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:11:50.972196 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "b8381811d2e449ce8d9ecb43b1449369"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "ed7cb033f9a7f6ef2fc459c7b9c4d97b"
server_key_iv: "8d6fdba344917fdb920d76aa84337e6d"
server_key_version: "encryptionkey@0"
I20260926 13:11:50.972543 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b8381811d2e449ce8d9ecb43b1449369"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "ed7cb033f9a7f6ef2fc459c7b9c4d97b"
server_key_iv: "8d6fdba344917fdb920d76aa84337e6d"
server_key_version: "encryptionkey@0"
I20260926 13:11:50.973922 15956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:11:50.974680 15971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:50.974881 15956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:50.975097 15956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "b8381811d2e449ce8d9ecb43b1449369"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "ed7cb033f9a7f6ef2fc459c7b9c4d97b"
server_key_iv: "8d6fdba344917fdb920d76aa84337e6d"
server_key_version: "encryptionkey@0"
I20260926 13:11:50.975170 15956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:50.996299 15956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:50.998389 15956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:50.998528 15956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:50.998757 15956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:50.999086 15956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:50.999121 15956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:50.999150 15956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:50.999167 15956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:51.004722 15956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43641
I20260926 13:11:51.004777 16084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43641 every 8 connection(s)
I20260926 13:11:51.005091 15956 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:11:51.009785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 15956
I20260926 13:11:51.009895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:11:51.010118 392 external_mini_cluster.cc:1428] Setting key c7569a19d38ddcc505ee73ed93eef351
Sep 26 13:11:51 dist-test-slave-2dw5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:51.015009 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.005824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48443 (local address 127.0.98.62:40789)
0926 13:11:51.005910 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:51.005912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:51.006128 (+ 216us) server_negotiation.cc:407] Connection header received
0926 13:11:51.006702 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:51.006703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:51.006715 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:51.006767 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:51.007761 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.008265 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.009250 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.009448 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.009512 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.012072 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:51.012074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:51.012074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:51.012082 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:51.013495 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:51.013923 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:51.013924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:51.013924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:51.013986 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:51.014290 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:51.014292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:51.014292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:51.014439 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:51.014511 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:51.014787 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 13:11:51.014910 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:11:51.015214 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.005741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:48443)
0926 13:11:51.005921 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:51.005966 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:51.006534 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:51.006877 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:51.006882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:51.007462 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:51.007661 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.007666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.008350 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.008352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.009161 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.009163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.009518 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.009519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.009634 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.010276 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:51.010286 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:51.011969 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:51.013618 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:51.013625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:51.013627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:51.013854 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:51.014123 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:51.014123 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:51.014123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:51.014227 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:51.014507 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:51.014513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:51.014661 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 13:11:51.014746 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:11:51.014918 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260926 13:11:51.015681 16085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40789
I20260926 13:11:51.015769 16085 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:51.015950 16085 heartbeater.cc:511] Master 127.0.98.62:40789 requested a full tablet report, sending...
I20260926 13:11:51.016322 15760 ts_manager.cc:194] Registered new tserver with Master: b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641)
I20260926 13:11:51.016724 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48443
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:11:51.037971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40789
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:11:51.118407 16090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:11:51.118650 16090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:11:51.118686 16090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:11:51.119925 16090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:11:51.119980 16090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:11:51.120018 16090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:11:51.121551 16090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:11:51.121824 16090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:11:51.122042 16090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:11:51.127261 16098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:51.127270 16096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:11:51.127270 16095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:11:51.127548 16090 server_base.cc:1034] running on GCE node
I20260926 13:11:51.127780 16090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:11:51.128017 16090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:11:51.129197 16090 hybrid_clock.cc:648] HybridClock initialized: now 1790428311129177 us; error 30 us; skew 500 ppm
Sep 26 13:11:51 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:11:51.131886 16090 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:11:51.132391 16090 webserver.cc:460] Webserver started at http://127.0.98.3:46281/ using document root <none> and password file <none>
I20260926 13:11:51.132604 16090 fs_manager.cc:359] Metadata directory not provided
I20260926 13:11:51.132656 16090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:11:51.132786 16090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:11:51.133836 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "88ddce8e365a4ab3bb950b5d3c023495"
format_stamp: "Formatted at 2026-09-26 13:11:51 on dist-test-slave-2dw5"
server_key: "43ae988bf1c3a9ffc82c75328e6457d7"
server_key_iv: "e648003bc547c8570e12e8df9b3d857b"
server_key_version: "encryptionkey@0"
I20260926 13:11:51.134166 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "88ddce8e365a4ab3bb950b5d3c023495"
format_stamp: "Formatted at 2026-09-26 13:11:51 on dist-test-slave-2dw5"
server_key: "43ae988bf1c3a9ffc82c75328e6457d7"
server_key_iv: "e648003bc547c8570e12e8df9b3d857b"
server_key_version: "encryptionkey@0"
I20260926 13:11:51.135452 16090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:11:51.136237 16105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:51.136404 16090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:11:51.136487 16090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "88ddce8e365a4ab3bb950b5d3c023495"
format_stamp: "Formatted at 2026-09-26 13:11:51 on dist-test-slave-2dw5"
server_key: "43ae988bf1c3a9ffc82c75328e6457d7"
server_key_iv: "e648003bc547c8570e12e8df9b3d857b"
server_key_version: "encryptionkey@0"
I20260926 13:11:51.136546 16090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:11:51.146324 16090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:11:51.148291 16090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:11:51.148423 16090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:11:51.148633 16090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:11:51.148913 16090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:11:51.148945 16090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:51.148972 16090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:11:51.148995 16090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:11:51.154867 16090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35135
I20260926 13:11:51.154927 16218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35135 every 8 connection(s)
I20260926 13:11:51.155280 16090 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:11:51 dist-test-slave-2dw5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:11:51.162834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 16090
I20260926 13:11:51.162963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:11:51.163206 392 external_mini_cluster.cc:1428] Setting key 6984b2a1dbe983d5e2065f18a44e7dfd
I20260926 13:11:51.164917 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.156007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42611 (local address 127.0.98.62:40789)
0926 13:11:51.156153 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:51.156155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:51.156248 (+ 93us) server_negotiation.cc:407] Connection header received
0926 13:11:51.156717 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:51.156718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:51.156727 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:51.156783 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:51.157783 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.158286 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.159456 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.159657 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.159723 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.161877 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:51.161878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:51.161879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:51.161885 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:51.163424 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:51.163916 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:51.163918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:51.163918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:51.163984 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:51.164277 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:51.164278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:51.164278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:51.164414 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:51.164466 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:51.164771 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 13:11:51.164851 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:11:51.165030 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.155970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:42611)
0926 13:11:51.156139 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:51.156182 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:51.156643 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:51.156836 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:51.156839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:51.157487 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:11:51.157680 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.157685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.158389 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.158390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.159363 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.159365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.159740 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.159741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.159845 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.160397 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:11:51.160405 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:11:51.161759 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:11:51.163514 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:51.163522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:51.163532 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:51.163845 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:51.164053 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:11:51.164054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:11:51.164054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:11:51.164203 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:11:51.164469 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:11:51.164474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:11:51.164626 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 13:11:51.164715 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:11:51.164850 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260926 13:11:51.165462 16219 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40789
I20260926 13:11:51.165557 16219 heartbeater.cc:464] Registering TS with master...
I20260926 13:11:51.165767 16219 heartbeater.cc:511] Master 127.0.98.62:40789 requested a full tablet report, sending...
I20260926 13:11:51.166162 15760 ts_manager.cc:194] Registered new tserver with Master: 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135)
I20260926 13:11:51.166544 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42611
I20260926 13:11:51.175452 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:11:51.182870 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.176857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.0.98.62:40789)
0926 13:11:51.176930 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:51.176933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:51.176974 (+ 41us) server_negotiation.cc:407] Connection header received
0926 13:11:51.177113 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:51.177114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:51.177125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:51.177173 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:11:51.177810 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.178347 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.179140 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.179325 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.179396 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.180503 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:11:51.180505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:11:51.180505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:11:51.180512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:11:51.181758 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:51.182102 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:51.182103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:51.182104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:51.182161 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:11:51.182322 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:11:51.182323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:11:51.182323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:11:51.182478 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:11:51.182563 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:51.182674 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 13:11:51.182751 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:11:51.184489 15760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:11:51.186259 15760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:11:51.192404 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.188809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.0.98.1:45747)
0926 13:11:51.189179 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:51.189183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:51.189191 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:11:51.189226 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:51.189228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:51.189241 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:51.189475 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:51.189538 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.190212 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.191519 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.192000 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.192067 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.192133 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:51.192276 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 13:11:51.192314 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":68,"threads_started":1}
I20260926 13:11:51.192394 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.188551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45747 (local address 127.0.0.1:45042)
0926 13:11:51.188837 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:51.188848 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:51.188917 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:51.189337 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:51.189338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:51.189340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:51.189404 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.189406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.190303 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.190305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.191441 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.191444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.192091 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.192092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.192179 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.192193 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 13:11:51.192253 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 13:11:51.192310 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"spinlock_wait_cycles":896,"thread_start_us":74,"threads_started":1}
I20260926 13:11:51.193244 15885 tablet_service.cc:1467] Processing CreateTablet for tablet 53efb15090a24c2692700cf57fadc843 (DEFAULT_TABLE table=test-table [id=d8ee93d8fe3a4e1cbcbe13e15f0f2990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:11:51.193624 15885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53efb15090a24c2692700cf57fadc843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:51.193763 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.188544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35135 (local address 127.0.0.1:42100)
0926 13:11:51.188809 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:51.188847 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:51.188918 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:51.190491 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:51.190492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:51.190494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:51.190543 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.190545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.191451 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.191453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.192575 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.192578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.193498 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.193500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.193567 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.193573 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:11:51.193635 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:11:51.193683 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":56,"threads_started":1}
I20260926 13:11:51.194080 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.188674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.0.98.3:35135)
0926 13:11:51.190359 (+ 1685us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:51.190363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:51.190371 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:11:51.190399 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:51.190401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:51.190412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:51.190467 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:51.190615 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.191364 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.192669 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.193357 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.193430 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.193487 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:51.193952 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 13:11:51.193998 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1651,"thread_start_us":81,"threads_started":1}
I20260926 13:11:51.194203 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.188553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43641 (local address 127.0.0.1:37970)
0926 13:11:51.188875 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:51.188883 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:51.188924 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:51.189317 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:51.189320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:51.189322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:51.189401 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.189404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.191531 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.191532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.192794 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.192797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.193947 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.193948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.194024 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.194028 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:11:51.194075 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 13:11:51.194121 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":62,"threads_started":1}
I20260926 13:11:51.194588 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.188982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37970 (local address 127.0.98.2:43641)
0926 13:11:51.189152 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:51.189157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:51.189168 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:11:51.189208 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:51.189210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:51.189226 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:51.189310 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:51.189495 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.190202 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.192900 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.193796 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.193862 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.193916 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:51.194108 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 13:11:51.194511 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260926 13:11:51.194778 16153 tablet_service.cc:1467] Processing CreateTablet for tablet 53efb15090a24c2692700cf57fadc843 (DEFAULT_TABLE table=test-table [id=d8ee93d8fe3a4e1cbcbe13e15f0f2990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:11:51.195143 16153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53efb15090a24c2692700cf57fadc843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:51.195261 16019 tablet_service.cc:1467] Processing CreateTablet for tablet 53efb15090a24c2692700cf57fadc843 (DEFAULT_TABLE table=test-table [id=d8ee93d8fe3a4e1cbcbe13e15f0f2990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:11:51.195597 16019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53efb15090a24c2692700cf57fadc843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:11:51.197145 16233 tablet_bootstrap.cc:492] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Bootstrap starting.
I20260926 13:11:51.197732 16234 tablet_bootstrap.cc:492] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495: Bootstrap starting.
I20260926 13:11:51.197849 16233 tablet_bootstrap.cc:654] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:51.198323 16233 log.cc:824] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:51.198395 16234 tablet_bootstrap.cc:654] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:51.198922 16236 tablet_bootstrap.cc:492] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Bootstrap starting.
I20260926 13:11:51.199015 16234 log.cc:824] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:51.199638 16236 tablet_bootstrap.cc:654] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Neither blocks nor log segments found. Creating new log.
I20260926 13:11:51.199744 16233 tablet_bootstrap.cc:492] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: No bootstrap required, opened a new log
I20260926 13:11:51.199795 16234 tablet_bootstrap.cc:492] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495: No bootstrap required, opened a new log
I20260926 13:11:51.199818 16233 ts_tablet_manager.cc:1397] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 13:11:51.199864 16234 ts_tablet_manager.cc:1397] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:11:51.200181 16236 log.cc:824] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Log is configured to *not* fsync() on all Append() calls
I20260926 13:11:51.201056 16236 tablet_bootstrap.cc:492] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: No bootstrap required, opened a new log
I20260926 13:11:51.201129 16236 ts_tablet_manager.cc:1397] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:11:51.201562 16233 raft_consensus.cc:348] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:11:51.201731 16233 raft_consensus.cc:374] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:51.201769 16233 raft_consensus.cc:729] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c423f60803b4469a93d739027ebe5450, State: Initialized, Role: FOLLOWER
I20260926 13:11:51.201766 16234 raft_consensus.cc:348] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:11:51.201936 16234 raft_consensus.cc:374] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:51.201917 16233 consensus_queue.cc:260] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:11:51.201978 16234 raft_consensus.cc:729] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ddce8e365a4ab3bb950b5d3c023495, State: Initialized, Role: FOLLOWER
I20260926 13:11:51.202078 16234 consensus_queue.cc:260] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:11:51.202219 16233 ts_tablet_manager.cc:1428] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 13:11:51.202296 15951 heartbeater.cc:503] Master 127.0.98.62:40789 was elected leader, sending a full tablet report...
I20260926 13:11:51.202351 16234 ts_tablet_manager.cc:1428] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 13:11:51.202476 16219 heartbeater.cc:503] Master 127.0.98.62:40789 was elected leader, sending a full tablet report...
I20260926 13:11:51.203469 16236 raft_consensus.cc:348] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:11:51.203665 16236 raft_consensus.cc:374] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:11:51.203703 16236 raft_consensus.cc:729] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8381811d2e449ce8d9ecb43b1449369, State: Initialized, Role: FOLLOWER
I20260926 13:11:51.203816 16236 consensus_queue.cc:260] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:11:51.204073 16236 ts_tablet_manager.cc:1428] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 13:11:51.204100 16085 heartbeater.cc:503] Master 127.0.98.62:40789 was elected leader, sending a full tablet report...
W20260926 13:11:51.255684 16086 tablet.cc:2369] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:11:51.306267 16240 raft_consensus.cc:482] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:11:51.306402 16240 raft_consensus.cc:504] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:11:51.306768 16240 leader_election.cc:288] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641), c423f60803b4469a93d739027ebe5450 (127.0.98.1:45747)
I20260926 13:11:51.310379 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.307088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41629 (local address 127.0.98.2:43641)
0926 13:11:51.307139 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:51.307142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:51.307152 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:11:51.307180 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:51.307181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:51.307192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:51.307238 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:51.307409 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.307963 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.309390 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.310006 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.310086 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.310098 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:51.310265 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 13:11:51.310307 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 13:11:51.310387 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.306990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43641 (local address 127.0.98.3:41629)
0926 13:11:51.307056 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:51.307068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:51.307118 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:51.307260 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:51.307260 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:51.307262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:51.307344 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.307346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.308028 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.308029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.309279 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.309282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.310097 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.310099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.310187 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.310191 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:11:51.310247 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 13:11:51.310295 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":25}
I20260926 13:11:51.310420 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.307062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58423 (local address 127.0.98.1:45747)
0926 13:11:51.307111 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:51.307113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:51.307247 (+ 134us) server_negotiation.cc:407] Connection header received
0926 13:11:51.307375 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:51.307377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:51.307388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:51.307434 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:11:51.307575 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.308178 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.309434 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.310009 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.310188 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.310197 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:51.310322 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 13:11:51.310364 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:11:51.310400 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.306991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45747 (local address 127.0.98.3:58423)
0926 13:11:51.307177 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:51.307189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:11:51.307274 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:11:51.307446 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:11:51.307447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:11:51.307449 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:11:51.307516 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.307518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.308242 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.308243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.309341 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:11:51.309343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.310192 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.310193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:11:51.310261 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.310263 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:11:51.310306 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 13:11:51.310335 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":33,"thread_start_us":124,"threads_started":1}
I20260926 13:11:51.310817 15905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53efb15090a24c2692700cf57fadc843" candidate_uuid: "88ddce8e365a4ab3bb950b5d3c023495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c423f60803b4469a93d739027ebe5450" is_pre_election: true
I20260926 13:11:51.310817 16039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53efb15090a24c2692700cf57fadc843" candidate_uuid: "88ddce8e365a4ab3bb950b5d3c023495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8381811d2e449ce8d9ecb43b1449369" is_pre_election: true
I20260926 13:11:51.310994 15905 raft_consensus.cc:2454] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88ddce8e365a4ab3bb950b5d3c023495 in term 0.
I20260926 13:11:51.311021 16039 raft_consensus.cc:2454] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88ddce8e365a4ab3bb950b5d3c023495 in term 0.
I20260926 13:11:51.311221 16107 leader_election.cc:302] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88ddce8e365a4ab3bb950b5d3c023495, c423f60803b4469a93d739027ebe5450; no voters:
I20260926 13:11:51.311386 16240 raft_consensus.cc:2781] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:11:51.311441 16240 raft_consensus.cc:482] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:11:51.311463 16240 raft_consensus.cc:3037] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:51.312258 16240 raft_consensus.cc:504] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:11:51.312407 16240 leader_election.cc:288] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [CANDIDATE]: Term 1 election: Requested vote from peers b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641), c423f60803b4469a93d739027ebe5450 (127.0.98.1:45747)
I20260926 13:11:51.312618 16039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53efb15090a24c2692700cf57fadc843" candidate_uuid: "88ddce8e365a4ab3bb950b5d3c023495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8381811d2e449ce8d9ecb43b1449369"
I20260926 13:11:51.312610 15905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53efb15090a24c2692700cf57fadc843" candidate_uuid: "88ddce8e365a4ab3bb950b5d3c023495" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c423f60803b4469a93d739027ebe5450"
I20260926 13:11:51.312698 16039 raft_consensus.cc:3037] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:51.312700 15905 raft_consensus.cc:3037] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:11:51.313438 16039 raft_consensus.cc:2454] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88ddce8e365a4ab3bb950b5d3c023495 in term 1.
I20260926 13:11:51.313593 15905 raft_consensus.cc:2454] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88ddce8e365a4ab3bb950b5d3c023495 in term 1.
I20260926 13:11:51.313616 16108 leader_election.cc:302] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88ddce8e365a4ab3bb950b5d3c023495, b8381811d2e449ce8d9ecb43b1449369; no voters:
I20260926 13:11:51.313719 16240 raft_consensus.cc:2781] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:11:51.313872 16240 raft_consensus.cc:686] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 1 LEADER]: Becoming Leader. State: Replica: 88ddce8e365a4ab3bb950b5d3c023495, State: Running, Role: LEADER
I20260926 13:11:51.313971 16240 consensus_queue.cc:237] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:11:51.314978 15759 catalog_manager.cc:5495] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "88ddce8e365a4ab3bb950b5d3c023495" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } health_report { overall_health: HEALTHY } } }
I20260926 13:11:51.342326 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.339389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.0.98.3:35135)
0926 13:11:51.339442 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:11:51.339444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:11:51.339454 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:11:51.339510 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:11:51.339511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:11:51.339535 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:11:51.339625 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:11:51.339774 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.340515 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.341586 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:11:51.341847 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:11:51.341899 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:11:51.342063 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:11:51.342164 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:11:51.342213 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:11:51.342229 (+ 16us) server_negotiation.cc:299] Negotiation successful
0926 13:11:51.342247 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:11:51.344292 15905 raft_consensus.cc:1261] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Refusing update from remote peer 88ddce8e365a4ab3bb950b5d3c023495: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:11:51.344318 16039 raft_consensus.cc:1261] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 1 FOLLOWER]: Refusing update from remote peer 88ddce8e365a4ab3bb950b5d3c023495: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:11:51.344643 16243 consensus_queue.cc:1035] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:11:51.344777 16240 consensus_queue.cc:1035] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 13:11:51.344875 15952 tablet.cc:2369] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:11:51.347604 16247 mvcc.cc:205] Tried to move back new op lower bound from 7333594363263188992 to 7333594363142701056. Current Snapshot: MvccSnapshot[applied={T|T < 7333594363263188992}]
I20260926 13:11:51.349195 15760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:11:51.349287 15760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:11:51.349977 15760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Sending DeleteTablet for 3 replicas of tablet 53efb15090a24c2692700cf57fadc843
I20260926 13:11:51.350317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 15822
I20260926 13:11:51.350303 15885 tablet_service.cc:1514] Processing DeleteTablet for tablet 53efb15090a24c2692700cf57fadc843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45042
I20260926 13:11:51.350358 16019 tablet_service.cc:1514] Processing DeleteTablet for tablet 53efb15090a24c2692700cf57fadc843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37970
I20260926 13:11:51.350579 16153 tablet_service.cc:1514] Processing DeleteTablet for tablet 53efb15090a24c2692700cf57fadc843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42100
I20260926 13:11:51.351326 16257 tablet_replica.cc:330] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495: stopping tablet replica
I20260926 13:11:51.351420 16257 raft_consensus.cc:2229] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:11:51.351533 16257 raft_consensus.cc:2258] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:11:51.351956 16257 ts_tablet_manager.cc:1905] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:11:51.353592 16257 ts_tablet_manager.cc:1918] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:11:51.353654 16257 log.cc:1196] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/53efb15090a24c2692700cf57fadc843
I20260926 13:11:51.353885 16257 ts_tablet_manager.cc:1939] T 53efb15090a24c2692700cf57fadc843 P 88ddce8e365a4ab3bb950b5d3c023495: Deleting consensus metadata
I20260926 13:11:51.354962 15747 catalog_manager.cc:4841] TS 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135): tablet 53efb15090a24c2692700cf57fadc843 (table test-table [id=d8ee93d8fe3a4e1cbcbe13e15f0f2990]) successfully deleted
I20260926 13:11:51.355427 16256 tablet_replica.cc:330] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: stopping tablet replica
I20260926 13:11:51.355577 16256 raft_consensus.cc:2229] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:11:51.355700 16256 raft_consensus.cc:2258] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 13:11:51.356114 15745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45747 (error 108)
I20260926 13:11:51.356187 16256 ts_tablet_manager.cc:1905] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:11:51.356341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 15956
I20260926 13:11:51.360761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 16090
I20260926 13:11:51.360956 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.360818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45747 (local address 127.0.0.1:45046)
0926 13:11:51.360883 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:51.360922 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 13:11:51.362123 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:11:51.362005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43641 (local address 127.0.0.1:37980)
0926 13:11:51.362058 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 13:11:51.362088 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 13:11:51.362284 15746 catalog_manager.cc:4584] TS b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53efb15090a24c2692700cf57fadc843: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43641: connect: Connection refused (error 111)
I20260926 13:11:51.365301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 15728
I20260926 13:12:11.370498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40789
--webserver_interface=127.0.98.62
--webserver_port=42261
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:11.459704 16259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:11.459964 16259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:11.459993 16259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:11.461202 16259 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 13:12:11.461241 16259 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 13:12:11.461257 16259 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 13:12:11.461272 16259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:12:11.461288 16259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:11.461315 16259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:11.461338 16259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:12:11.461361 16259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:12:11.462965 16259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:11.463238 16259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:11.463461 16259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:11.473234 16268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:11.473253 16266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:11.473475 16259 server_base.cc:1034] running on GCE node
W20260926 13:12:11.473248 16265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:11.473906 16259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:11.474331 16259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:11.475495 16259 hybrid_clock.cc:648] HybridClock initialized: now 1790428331475471 us; error 48 us; skew 500 ppm
Sep 26 13:12:11 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:11.478590 16259 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:12:11.479175 16259 webserver.cc:460] Webserver started at http://127.0.98.62:42261/ using document root <none> and password file <none>
I20260926 13:12:11.479425 16259 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:11.479494 16259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:11.481276 16259 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:12:11.482028 16275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:11.482344 16259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:11.482462 16259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "3344c043df7e46ef9bbfe07db4a7c943"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "d6b61fb64de8d84cde4cf7ceb89b9bbc"
server_key_iv: "05897514493cd8878fd3b6e324ddc5d6"
server_key_version: "encryptionkey@0"
I20260926 13:12:11.483014 16259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:11.500087 16259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:11.502509 16259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:11.502664 16259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:11.507691 16259 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40789
I20260926 13:12:11.507787 16327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40789 every 8 connection(s)
I20260926 13:12:11.508196 16259 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:12:11.510057 16328 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:12:11.510816 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Bootstrap starting.
I20260926 13:12:11.516455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 16259
I20260926 13:12:11.516808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45747
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40789
--webserver_port=42721
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:12:11.520282 16328 log.cc:824] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:11.522305 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:12:11.522578 16328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Bootstrap complete.
I20260926 13:12:11.524893 16328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260926 13:12:11.525152 16328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3344c043df7e46ef9bbfe07db4a7c943, State: Initialized, Role: FOLLOWER
I20260926 13:12:11.525316 16328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260926 13:12:11.525393 16328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:12:11.525440 16328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:12:11.525489 16328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:12:11.526578 16328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260926 13:12:11.526679 16328 leader_election.cc:302] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3344c043df7e46ef9bbfe07db4a7c943; no voters:
I20260926 13:12:11.526873 16328 leader_election.cc:288] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:12:11.526997 16332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:12:11.527176 16328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:12:11.527223 16332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 2 LEADER]: Becoming Leader. State: Replica: 3344c043df7e46ef9bbfe07db4a7c943, State: Running, Role: LEADER
I20260926 13:12:11.527356 16332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260926 13:12:11.527752 16333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3344c043df7e46ef9bbfe07db4a7c943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } } }
I20260926 13:12:11.527868 16333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:11.528038 16334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3344c043df7e46ef9bbfe07db4a7c943. Latest consensus state: current_term: 2 leader_uuid: "3344c043df7e46ef9bbfe07db4a7c943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } } }
I20260926 13:12:11.528093 16334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:11.528082 16341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:12:11.529148 16341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:12:11.529250 16341 catalog_manager.cc:1259] Loaded cluster ID: be9185f14b4040dbae12cda1d987e95a
I20260926 13:12:11.529291 16341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:12:11.530892 16341 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:12:11.531091 16341 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Loaded TSK: 0
I20260926 13:12:11.542984 16341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Generated new TSK 1
I20260926 13:12:11.543113 16341 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:12:11.613672 16330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:11.613932 16330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:11.613961 16330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:11.615365 16330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:11.615415 16330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:11.615446 16330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:12:11.617035 16330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42721
--tserver_master_addrs=127.0.98.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:11.617319 16330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:11.617576 16330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:11.623229 16356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:11.623224 16358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:11.623216 16355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:11.624490 16330 server_base.cc:1034] running on GCE node
I20260926 13:12:11.624764 16330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:11.625061 16330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:11.626257 16330 hybrid_clock.cc:648] HybridClock initialized: now 1790428331626244 us; error 32 us; skew 500 ppm
Sep 26 13:12:11 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:11.629432 16330 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:12:11.630007 16330 webserver.cc:460] Webserver started at http://127.0.98.1:42721/ using document root <none> and password file <none>
I20260926 13:12:11.630236 16330 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:11.630307 16330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:11.631945 16330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:11.632695 16365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:11.632853 16330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:11.632970 16330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "c423f60803b4469a93d739027ebe5450"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "701ab8d8eff39f333cd700d7ef95e70e"
server_key_iv: "cf77ab6ea847bec8790a3388b2dad95f"
server_key_version: "encryptionkey@0"
I20260926 13:12:11.633347 16330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:11.649852 16330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:11.652253 16330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:11.652410 16330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:11.652660 16330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:11.653144 16372 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:12:11.654754 16330 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:12:11.654805 16330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 13:12:11.654831 16330 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:12:11.655557 16330 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:12:11.655601 16330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 13:12:11.655699 16372 tablet_bootstrap.cc:492] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Bootstrap starting.
I20260926 13:12:11.663298 16330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45747
I20260926 13:12:11.663357 16479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45747 every 8 connection(s)
I20260926 13:12:11.663846 16330 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:12:11.667407 16372 log.cc:824] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:11.670176 16372 tablet_bootstrap.cc:492] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:12:11.670477 16372 tablet_bootstrap.cc:492] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Bootstrap complete.
I20260926 13:12:11.671554 16372 ts_tablet_manager.cc:1397] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
Sep 26 13:12:11 dist-test-slave-2dw5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:11.672719 16372 raft_consensus.cc:348] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:12:11.672999 16372 raft_consensus.cc:729] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c423f60803b4469a93d739027ebe5450, State: Initialized, Role: FOLLOWER
I20260926 13:12:11.673166 16372 consensus_queue.cc:260] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:12:11.673200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 16330
I20260926 13:12:11.673480 16372 ts_tablet_manager.cc:1428] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:12:11.673491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43641
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40789
--webserver_port=43221
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:12:11.676597 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.664694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52073 (local address 127.0.98.62:40789)
0926 13:12:11.664857 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:11.664865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:11.664904 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:12:11.665749 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:11.665755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:11.665797 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:11.665912 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:11.667038 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.668185 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.669784 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.670258 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.670377 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.672753 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:11.672758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:11.672758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:11.672769 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:11.674685 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:11.675150 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:11.675153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:11.675153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:11.675280 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:11.675505 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:11.675506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:11.675506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:11.675842 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:11.675936 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:11.676313 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 13:12:11.676431 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260926 13:12:11.676606 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.664589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:52073)
0926 13:12:11.664758 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:11.664807 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:11.665533 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:11.665995 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:11.666000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:11.666432 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:11.666772 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:11.666781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.668328 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.668329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:11.669656 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:11.669659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.670364 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.670365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:11.670520 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.671037 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:11.671055 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:11.672628 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:11.674796 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:11.674803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:11.674805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:11.675075 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:11.675345 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:11.675346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:11.675347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:11.675443 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:11.675970 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:11.675977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:11.676132 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 13:12:11.676233 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 13:12:11.676414 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260926 13:12:11.677320 16480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40789
I20260926 13:12:11.677428 16480 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:11.677608 16480 heartbeater.cc:511] Master 127.0.98.62:40789 requested a full tablet report, sending...
I20260926 13:12:11.678381 16292 ts_manager.cc:194] Registered new tserver with Master: c423f60803b4469a93d739027ebe5450 (127.0.98.1:45747)
I20260926 13:12:11.679719 16292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52073
I20260926 13:12:11.680706 16480 heartbeater.cc:503] Master 127.0.98.62:40789 was elected leader, sending a full tablet report...
Sep 26 13:12:11 dist-test-slave-2dw5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:12:11.687597 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.679188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.0.98.1:45747)
0926 13:12:11.679531 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:11.679535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:11.679541 (+ 6us) server_negotiation.cc:407] Connection header received
0926 13:12:11.679569 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:11.679571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:11.679588 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:11.679648 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:11.680118 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.681071 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.682143 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.682423 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.682478 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.684742 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:11.684750 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:11.684751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:11.684757 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:11.686255 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:11.686703 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:11.686706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:11.686707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:11.686767 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:11.687005 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:11.687006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:11.687006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:11.687173 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:11.687238 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:11.687411 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 13:12:11.687512 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":71,"threads_started":1}
I20260926 13:12:11.687649 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.679120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45747 (local address 127.0.0.1:34862)
0926 13:12:11.679324 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:11.679349 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:11.679401 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:11.679663 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:11.679669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:11.679979 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:11.680036 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:11.680039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.681150 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.681151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:11.682069 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:11.682072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.682495 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.682500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:11.682583 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.683072 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:11.683077 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:11.684651 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:11.686325 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:11.686332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:11.686335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:11.686607 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:11.686872 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:11.686873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:11.686874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:11.686937 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:11.687233 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:11.687239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:11.687296 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 13:12:11.687388 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:12:11.687527 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260926 13:12:11.688043 16414 tablet_service.cc:1514] Processing DeleteTablet for tablet 53efb15090a24c2692700cf57fadc843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34862
I20260926 13:12:11.688685 16489 tablet_replica.cc:330] stopping tablet replica
I20260926 13:12:11.688799 16489 raft_consensus.cc:2229] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:11.688874 16489 raft_consensus.cc:2258] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:11.689172 16489 ts_tablet_manager.cc:1905] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:11.691416 16489 ts_tablet_manager.cc:1918] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:11.691480 16489 log.cc:1196] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal/wals/53efb15090a24c2692700cf57fadc843
I20260926 13:12:11.691674 16489 ts_tablet_manager.cc:1939] T 53efb15090a24c2692700cf57fadc843 P c423f60803b4469a93d739027ebe5450: Deleting consensus metadata
I20260926 13:12:11.692159 16277 catalog_manager.cc:4841] TS c423f60803b4469a93d739027ebe5450 (127.0.98.1:45747): tablet 53efb15090a24c2692700cf57fadc843 (table test-table [id=d8ee93d8fe3a4e1cbcbe13e15f0f2990]) successfully deleted
W20260926 13:12:11.764809 16486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:11.765046 16486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:11.765079 16486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:11.766289 16486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:11.766332 16486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:11.766359 16486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:12:11.767949 16486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43221
--tserver_master_addrs=127.0.98.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:11.768213 16486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:11.768419 16486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:11.773541 16494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:11.773648 16495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:11.773578 16497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:11.773777 16486 server_base.cc:1034] running on GCE node
I20260926 13:12:11.774040 16486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:11.774293 16486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:11.775445 16486 hybrid_clock.cc:648] HybridClock initialized: now 1790428331775428 us; error 31 us; skew 500 ppm
Sep 26 13:12:11 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:11.778126 16486 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:12:11.778601 16486 webserver.cc:460] Webserver started at http://127.0.98.2:43221/ using document root <none> and password file <none>
I20260926 13:12:11.778855 16486 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:11.778939 16486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:11.780365 16486 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:11.781163 16504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:11.781339 16486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:11.781424 16486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "b8381811d2e449ce8d9ecb43b1449369"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "ed7cb033f9a7f6ef2fc459c7b9c4d97b"
server_key_iv: "8d6fdba344917fdb920d76aa84337e6d"
server_key_version: "encryptionkey@0"
I20260926 13:12:11.781767 16486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:11.791782 16486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:11.794050 16486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:11.794193 16486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:11.794412 16486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:11.794934 16511 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:12:11.796399 16486 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:12:11.796459 16486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 13:12:11.796495 16486 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:12:11.797267 16486 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:12:11.797320 16486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 13:12:11.797412 16511 tablet_bootstrap.cc:492] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Bootstrap starting.
I20260926 13:12:11.799165 16511 log.cc:824] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:11.802415 16511 tablet_bootstrap.cc:492] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:12:11.802735 16511 tablet_bootstrap.cc:492] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Bootstrap complete.
I20260926 13:12:11.802927 16511 ts_tablet_manager.cc:1397] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20260926 13:12:11.803858 16486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43641
I20260926 13:12:11.803939 16619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43641 every 8 connection(s)
I20260926 13:12:11.804255 16486 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:12:11.804863 16511 raft_consensus.cc:348] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:12:11.805228 16511 raft_consensus.cc:729] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8381811d2e449ce8d9ecb43b1449369, State: Initialized, Role: FOLLOWER
I20260926 13:12:11.805363 16511 consensus_queue.cc:260] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } } peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } }
I20260926 13:12:11.805754 16511 ts_tablet_manager.cc:1428] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 13:12:11.809397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 16486
I20260926 13:12:11.809664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35135
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40789
--webserver_port=46281
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:12:11 dist-test-slave-2dw5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:11.813503 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.805019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52285 (local address 127.0.98.62:40789)
0926 13:12:11.805082 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:11.805085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:11.805242 (+ 157us) server_negotiation.cc:407] Connection header received
0926 13:12:11.805432 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:11.805433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:11.805449 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:11.805527 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:11.806258 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.806800 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.807951 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.808171 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.808229 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.810675 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:11.810677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:11.810677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:11.810685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:11.812007 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:11.812447 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:11.812447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:11.812448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:11.812500 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:11.812774 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:11.812775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:11.812775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:11.812940 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:11.813017 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:11.813359 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 13:12:11.813436 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:12:11.813724 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.804947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:52285)
0926 13:12:11.805120 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:11.805162 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:11.805341 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:11.805568 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:11.805571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:11.805958 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:11.806156 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:11.806161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.807000 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.807001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:11.807876 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:11.807878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.808241 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.808242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:11.808348 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.809262 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:11.809270 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:11.810586 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:11.812095 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:11.812103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:11.812105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:11.812380 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:11.812550 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:11.812551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:11.812551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:11.812680 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:11.813021 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:11.813027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:11.813202 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 13:12:11.813322 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 13:12:11.813501 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260926 13:12:11.814265 16620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40789
I20260926 13:12:11.814354 16620 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:11.814520 16620 heartbeater.cc:511] Master 127.0.98.62:40789 requested a full tablet report, sending...
I20260926 13:12:11.815042 16292 ts_manager.cc:194] Registered new tserver with Master: b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641)
I20260926 13:12:11.815585 16292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52285
I20260926 13:12:11.816347 16620 heartbeater.cc:503] Master 127.0.98.62:40789 was elected leader, sending a full tablet report...
Sep 26 13:12:11 dist-test-slave-2dw5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 13:12:11.822566 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.815423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.0.98.2:43641)
0926 13:12:11.815564 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:11.815568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:11.815577 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:12:11.815611 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:11.815612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:11.815626 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:11.815835 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:11.816139 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.817214 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.817866 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.818331 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.818405 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.820203 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:11.820210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:11.820210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:11.820215 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:11.821556 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:11.821901 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:11.821904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:11.821904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:11.821950 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:11.822127 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:11.822128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:11.822128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:11.822253 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:11.822311 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:11.822424 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 13:12:11.822493 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260926 13:12:11.822603 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.815356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43641 (local address 127.0.0.1:37574)
0926 13:12:11.815407 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:11.815418 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:11.815453 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:11.815850 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:11.815851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:11.816005 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:11.816048 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:11.816051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.817298 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.817299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:11.817792 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:11.817793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.818400 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.818402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:11.818448 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.818889 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:11.818893 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:11.820109 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:11.821628 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:11.821629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:11.821630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:11.821838 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:11.821997 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:11.821998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:11.821998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:11.822054 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:11.822303 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:11.822303 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:11.822346 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 13:12:11.822406 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 13:12:11.822508 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 13:12:11.823088 16554 tablet_service.cc:1514] Processing DeleteTablet for tablet 53efb15090a24c2692700cf57fadc843 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:11:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37574
I20260926 13:12:11.823801 16626 tablet_replica.cc:330] stopping tablet replica
I20260926 13:12:11.823899 16626 raft_consensus.cc:2229] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:11.823966 16626 raft_consensus.cc:2258] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:11.824277 16626 ts_tablet_manager.cc:1905] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:11.826690 16626 ts_tablet_manager.cc:1918] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:11.826756 16626 log.cc:1196] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/53efb15090a24c2692700cf57fadc843
I20260926 13:12:11.826972 16626 ts_tablet_manager.cc:1939] T 53efb15090a24c2692700cf57fadc843 P b8381811d2e449ce8d9ecb43b1449369: Deleting consensus metadata
I20260926 13:12:11.827517 16278 catalog_manager.cc:4841] TS b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641): tablet 53efb15090a24c2692700cf57fadc843 (table test-table [id=d8ee93d8fe3a4e1cbcbe13e15f0f2990]) successfully deleted
W20260926 13:12:11.893204 16624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:11.893426 16624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:11.893457 16624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:11.894583 16624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:11.894626 16624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:11.894661 16624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:12:11.896162 16624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46281
--tserver_master_addrs=127.0.98.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:11.896418 16624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:11.896631 16624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:12:11.902007 16624 server_base.cc:1034] running on GCE node
W20260926 13:12:11.902081 16634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:11.902153 16632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:11.902084 16631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:11.902410 16624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:11.902664 16624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:11.903806 16624 hybrid_clock.cc:648] HybridClock initialized: now 1790428331903790 us; error 33 us; skew 500 ppm
Sep 26 13:12:11 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:11.906230 16624 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:12:11.906735 16624 webserver.cc:460] Webserver started at http://127.0.98.3:46281/ using document root <none> and password file <none>
I20260926 13:12:11.906984 16624 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:11.907049 16624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:11.908421 16624 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:11.909116 16641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:11.909297 16624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:11.909452 16624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "88ddce8e365a4ab3bb950b5d3c023495"
format_stamp: "Formatted at 2026-09-26 13:11:51 on dist-test-slave-2dw5"
server_key: "43ae988bf1c3a9ffc82c75328e6457d7"
server_key_iv: "e648003bc547c8570e12e8df9b3d857b"
server_key_version: "encryptionkey@0"
I20260926 13:12:11.909811 16624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:11.930635 16624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:11.932850 16624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:11.932992 16624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:11.933216 16624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:11.933511 16624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:11.933542 16624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:11.933573 16624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:11.933593 16624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:11.939471 16624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35135
I20260926 13:12:11.939522 16754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35135 every 8 connection(s)
I20260926 13:12:11.939869 16624 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:12:11.944916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 16624
Sep 26 13:12:11 dist-test-slave-2dw5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:11.949493 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.940644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57451 (local address 127.0.98.62:40789)
0926 13:12:11.940694 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:11.940696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:11.940862 (+ 166us) server_negotiation.cc:407] Connection header received
0926 13:12:11.941319 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:11.941320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:11.941330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:11.941389 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:11.942299 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.942795 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.943768 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.943956 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.944008 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.946552 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:11.946554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:11.946554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:11.946562 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:11.948051 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:11.948528 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:11.948529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:11.948530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:11.948586 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:11.948829 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:11.948830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:11.948831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:11.948970 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:11.949034 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:11.949313 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 13:12:11.949401 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:12:11.949681 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.940597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:57451)
0926 13:12:11.940746 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:11.940787 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:11.941241 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:11.941450 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:11.941453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:11.942013 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:11.942206 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:11.942210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.942911 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.942911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:11.943703 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:11.943705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.944033 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.944034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:11.944149 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.944659 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:11.944667 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:11.946367 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:11.948142 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:11.948152 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:11.948154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:11.948467 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:11.948644 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:11.948645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:11.948645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:11.948757 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:11.949040 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:11.949047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:11.949192 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 13:12:11.949284 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:12:11.949460 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260926 13:12:11.950151 16755 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40789
I20260926 13:12:11.950248 16755 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:11.950424 16755 heartbeater.cc:511] Master 127.0.98.62:40789 requested a full tablet report, sending...
I20260926 13:12:11.950762 16292 ts_manager.cc:194] Registered new tserver with Master: 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135)
I20260926 13:12:11.951149 16292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57451
I20260926 13:12:11.951670 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.945367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47668 (local address 127.0.98.62:40789)
0926 13:12:11.945485 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:11.945486 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:11.945547 (+ 61us) server_negotiation.cc:407] Connection header received
0926 13:12:11.945650 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:11.945651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:11.945660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:11.945711 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:11.946449 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.947055 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.947774 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.947993 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.948048 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.949088 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:11.949089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:11.949090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:11.949096 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:11.950529 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:11.950904 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:11.950905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:11.950906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:11.950961 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:11.951176 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:11.951176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:11.951177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:11.951317 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:11.951463 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:11.951500 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 13:12:11.951581 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260926 13:12:11.952344 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:12:11.955214 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.952682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.0.98.62:40789)
0926 13:12:11.952724 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:11.952725 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:11.952828 (+ 103us) server_negotiation.cc:407] Connection header received
0926 13:12:11.952876 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:11.952877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:11.952903 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:11.952957 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:12:11.953043 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.953616 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.954330 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.954609 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.954664 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.954881 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:12:11.954992 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:12:11.955058 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:11.955145 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 13:12:11.955164 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:12:11.955482 16292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:12:11.957106 16292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:12:11.959089 16554 tablet_service.cc:1467] Processing CreateTablet for tablet 95f8b17655d644e784773727827144bc (DEFAULT_TABLE table=test-table [id=57bcab0f768a4a1c940611ecd6eae3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:11.959261 16554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f8b17655d644e784773727827144bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:11.960680 16511 tablet_bootstrap.cc:492] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369: Bootstrap starting.
I20260926 13:12:11.960944 16414 tablet_service.cc:1467] Processing CreateTablet for tablet 95f8b17655d644e784773727827144bc (DEFAULT_TABLE table=test-table [id=57bcab0f768a4a1c940611ecd6eae3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:11.961095 16511 tablet_bootstrap.cc:654] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:11.961105 16414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f8b17655d644e784773727827144bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:11.961625 16511 tablet_bootstrap.cc:492] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369: No bootstrap required, opened a new log
I20260926 13:12:11.961668 16511 ts_tablet_manager.cc:1397] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 13:12:11.961948 16511 raft_consensus.cc:348] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:11.962033 16511 raft_consensus.cc:374] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:11.962083 16511 raft_consensus.cc:729] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8381811d2e449ce8d9ecb43b1449369, State: Initialized, Role: FOLLOWER
I20260926 13:12:11.962128 16511 consensus_queue.cc:260] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:11.962212 16511 ts_tablet_manager.cc:1428] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:12:11.962569 16372 tablet_bootstrap.cc:492] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450: Bootstrap starting.
I20260926 13:12:11.962865 16372 tablet_bootstrap.cc:654] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:11.963683 16372 tablet_bootstrap.cc:492] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450: No bootstrap required, opened a new log
I20260926 13:12:11.963742 16372 ts_tablet_manager.cc:1397] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 13:12:11.963873 16372 raft_consensus.cc:348] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:11.963929 16372 raft_consensus.cc:374] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:11.963969 16372 raft_consensus.cc:729] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c423f60803b4469a93d739027ebe5450, State: Initialized, Role: FOLLOWER
I20260926 13:12:11.964018 16372 consensus_queue.cc:260] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:11.964115 16372 ts_tablet_manager.cc:1428] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 13:12:11.965775 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.958935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35135 (local address 127.0.0.1:42120)
0926 13:12:11.960633 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:11.960643 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:11.960690 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:11.962743 (+ 2053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:11.962744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:11.962746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:11.962796 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:11.962798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.963759 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.963760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:11.964808 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:11.964812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.965513 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.965514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:11.965578 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.965597 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:12:11.965653 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 13:12:11.965701 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1680}
I20260926 13:12:11.965788 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:11.962373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42120 (local address 127.0.98.3:35135)
0926 13:12:11.962576 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:11.962581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:11.962592 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:12:11.962640 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:11.962642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:11.962658 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:11.962744 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:11.962884 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.963681 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.964902 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:11.965436 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:11.965499 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:11.965562 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:11.965669 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 13:12:11.965708 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":107,"threads_started":1}
I20260926 13:12:11.966445 16689 tablet_service.cc:1467] Processing CreateTablet for tablet 95f8b17655d644e784773727827144bc (DEFAULT_TABLE table=test-table [id=57bcab0f768a4a1c940611ecd6eae3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:11.966725 16689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f8b17655d644e784773727827144bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:11.969640 16764 tablet_bootstrap.cc:492] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495: Bootstrap starting.
I20260926 13:12:11.970250 16764 tablet_bootstrap.cc:654] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:11.970778 16764 log.cc:824] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:11.971560 16764 tablet_bootstrap.cc:492] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495: No bootstrap required, opened a new log
I20260926 13:12:11.971626 16764 ts_tablet_manager.cc:1397] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:12:11.972971 16764 raft_consensus.cc:348] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:11.973126 16764 raft_consensus.cc:374] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:11.973158 16764 raft_consensus.cc:729] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ddce8e365a4ab3bb950b5d3c023495, State: Initialized, Role: FOLLOWER
I20260926 13:12:11.973279 16764 consensus_queue.cc:260] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:11.973532 16764 ts_tablet_manager.cc:1428] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:12:11.973562 16755 heartbeater.cc:503] Master 127.0.98.62:40789 was elected leader, sending a full tablet report...
W20260926 13:12:12.054905 16621 tablet.cc:2369] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:12:12.061918 16623 raft_consensus.cc:482] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:12.062077 16623 raft_consensus.cc:504] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:12.062398 16623 leader_election.cc:288] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135), c423f60803b4469a93d739027ebe5450 (127.0.98.1:45747)
I20260926 13:12:12.066494 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:12.062627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45747 (local address 127.0.98.2:50725)
0926 13:12:12.062667 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:12.062680 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:12.062745 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:12.062955 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:12.062956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:12.062958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:12.063033 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:12.063035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.064024 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.064025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:12.065363 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:12.065368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.066165 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.066167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:12.066253 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:12.066287 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 13:12:12.066355 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 13:12:12.066408 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16,"mutex_wait_us":53}
I20260926 13:12:12.066519 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:12.062732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33271 (local address 127.0.98.3:35135)
0926 13:12:12.062772 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:12.062774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:12.062908 (+ 134us) server_negotiation.cc:407] Connection header received
0926 13:12:12.062976 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:12.062977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:12.062989 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:12.063026 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:12.063184 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.063867 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.065476 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.066051 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.066143 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:12.066155 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:12.066399 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 13:12:12.066447 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:12:12.066520 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:12.062707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50725 (local address 127.0.98.1:45747)
0926 13:12:12.062754 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:12.062756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:12.062767 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:12:12.062823 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:12.062824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:12.062861 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:12.062933 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:12.063104 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.063933 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.065475 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.066089 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.066148 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:12.066221 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:12.066396 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 13:12:12.066453 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:12:12.066633 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:12.062645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35135 (local address 127.0.98.2:33271)
0926 13:12:12.062816 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:12.062825 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:12.062911 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:12.063047 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:12.063047 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:12.063049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:12.063111 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:12.063113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.063982 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.063983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:12.065364 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:12.065368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.066189 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.066190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:12.066253 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:12.066287 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 13:12:12.066355 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 13:12:12.066496 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":39,"thread_start_us":102,"threads_started":1}
I20260926 13:12:12.066994 16434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f8b17655d644e784773727827144bc" candidate_uuid: "b8381811d2e449ce8d9ecb43b1449369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c423f60803b4469a93d739027ebe5450" is_pre_election: true
I20260926 13:12:12.067052 16709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f8b17655d644e784773727827144bc" candidate_uuid: "b8381811d2e449ce8d9ecb43b1449369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ddce8e365a4ab3bb950b5d3c023495" is_pre_election: true
I20260926 13:12:12.067140 16434 raft_consensus.cc:2454] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8381811d2e449ce8d9ecb43b1449369 in term 0.
I20260926 13:12:12.067190 16709 raft_consensus.cc:2454] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8381811d2e449ce8d9ecb43b1449369 in term 0.
I20260926 13:12:12.067392 16506 leader_election.cc:302] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8381811d2e449ce8d9ecb43b1449369, c423f60803b4469a93d739027ebe5450; no voters:
I20260926 13:12:12.067529 16623 raft_consensus.cc:2781] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:12:12.067579 16623 raft_consensus.cc:482] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:12:12.067598 16623 raft_consensus.cc:3037] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:12.068331 16623 raft_consensus.cc:504] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:12.068467 16623 leader_election.cc:288] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [CANDIDATE]: Term 1 election: Requested vote from peers 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135), c423f60803b4469a93d739027ebe5450 (127.0.98.1:45747)
I20260926 13:12:12.068645 16709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f8b17655d644e784773727827144bc" candidate_uuid: "b8381811d2e449ce8d9ecb43b1449369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ddce8e365a4ab3bb950b5d3c023495"
I20260926 13:12:12.068645 16434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f8b17655d644e784773727827144bc" candidate_uuid: "b8381811d2e449ce8d9ecb43b1449369" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c423f60803b4469a93d739027ebe5450"
I20260926 13:12:12.068732 16434 raft_consensus.cc:3037] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:12.068732 16709 raft_consensus.cc:3037] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:12.069626 16709 raft_consensus.cc:2454] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8381811d2e449ce8d9ecb43b1449369 in term 1.
I20260926 13:12:12.069625 16434 raft_consensus.cc:2454] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8381811d2e449ce8d9ecb43b1449369 in term 1.
I20260926 13:12:12.069821 16508 leader_election.cc:302] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88ddce8e365a4ab3bb950b5d3c023495, b8381811d2e449ce8d9ecb43b1449369; no voters:
I20260926 13:12:12.069931 16623 raft_consensus.cc:2781] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:12.070098 16623 raft_consensus.cc:686] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 1 LEADER]: Becoming Leader. State: Replica: b8381811d2e449ce8d9ecb43b1449369, State: Running, Role: LEADER
I20260926 13:12:12.070163 16623 consensus_queue.cc:237] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:12.070991 16292 catalog_manager.cc:5495] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b8381811d2e449ce8d9ecb43b1449369" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } health_report { overall_health: UNKNOWN } } }
I20260926 13:12:12.079001 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:12.076742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37576 (local address 127.0.98.2:43641)
0926 13:12:12.076795 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:12.076797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:12.076805 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:12:12.076840 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:12.076841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:12.076869 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:12.076927 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:12:12.077031 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.077562 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.078261 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.078476 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.078529 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:12.078724 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:12:12.078778 (+ 54us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 13:12:12.078931 (+ 153us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37576: token expired
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 13:12:12.079070 16625 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37576: token expired
I20260926 13:12:12.084812 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:12.079340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.0.98.62:40789)
0926 13:12:12.079376 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:12.079378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:12.079387 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:12:12.079410 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:12.079410 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:12.079419 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:12.079460 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:12.079883 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.080387 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.081106 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.081301 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.081351 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:12.082533 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:12.082534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:12.082535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:12.082541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:12.083866 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:12.084197 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:12.084197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:12.084198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:12.084253 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:12.084402 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:12.084403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:12.084403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:12.084503 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:12.084573 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:12.084684 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 13:12:12.084747 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:12:12.087975 16625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:12.085823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.0.98.2:43641)
0926 13:12:12.085901 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:12.085903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:12.085911 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:12:12.085931 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:12.085932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:12.085939 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:12.086006 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:12:12.086178 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.086652 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.087386 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:12.087562 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:12.087611 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:12.087725 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:12:12.087815 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:12:12.087851 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:12.087891 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 13:12:12.087905 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:12:12.089463 16434 raft_consensus.cc:1261] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Refusing update from remote peer b8381811d2e449ce8d9ecb43b1449369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:12.089462 16709 raft_consensus.cc:1261] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495 [term 1 FOLLOWER]: Refusing update from remote peer b8381811d2e449ce8d9ecb43b1449369: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:12.089823 16623 consensus_queue.cc:1035] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:12.089970 16768 consensus_queue.cc:1035] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:12.092218 16771 mvcc.cc:205] Tried to move back new op lower bound from 7333594448235819008 to 7333594448159830016. Current Snapshot: MvccSnapshot[applied={T|T < 7333594448235819008}]
I20260926 13:12:12.092828 16772 mvcc.cc:205] Tried to move back new op lower bound from 7333594448235819008 to 7333594448159830016. Current Snapshot: MvccSnapshot[applied={T|T < 7333594448235819008}]
I20260926 13:12:12.093290 16774 mvcc.cc:205] Tried to move back new op lower bound from 7333594448235819008 to 7333594448159830016. Current Snapshot: MvccSnapshot[applied={T|T < 7333594448235819008}]
I20260926 13:12:12.094532 16291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:12.094623 16291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:12.095340 16291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Sending DeleteTablet for 3 replicas of tablet 95f8b17655d644e784773727827144bc
I20260926 13:12:12.095618 16414 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f8b17655d644e784773727827144bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34862
I20260926 13:12:12.095620 16689 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f8b17655d644e784773727827144bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42120
I20260926 13:12:12.095714 16554 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f8b17655d644e784773727827144bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37574
I20260926 13:12:12.095800 16489 tablet_replica.cc:330] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450: stopping tablet replica
I20260926 13:12:12.095851 16489 raft_consensus.cc:2229] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:12.095907 16489 raft_consensus.cc:2258] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:12.095929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 16330
I20260926 13:12:12.095986 16626 tablet_replica.cc:330] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369: stopping tablet replica
I20260926 13:12:12.096045 16626 raft_consensus.cc:2229] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:12:12.096117 16626 raft_consensus.cc:2258] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:12.096690 16626 ts_tablet_manager.cc:1905] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:12.097026 16780 tablet_replica.cc:330] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495: stopping tablet replica
I20260926 13:12:12.097131 16780 raft_consensus.cc:2229] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:12.097229 16780 raft_consensus.cc:2258] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:12.097934 16780 ts_tablet_manager.cc:1905] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:12.098726 16626 ts_tablet_manager.cc:1918] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:12.098789 16626 log.cc:1196] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/95f8b17655d644e784773727827144bc
I20260926 13:12:12.098999 16626 ts_tablet_manager.cc:1939] T 95f8b17655d644e784773727827144bc P b8381811d2e449ce8d9ecb43b1449369: Deleting consensus metadata
I20260926 13:12:12.099316 16278 catalog_manager.cc:4841] TS b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641): tablet 95f8b17655d644e784773727827144bc (table test-table [id=57bcab0f768a4a1c940611ecd6eae3ed]) successfully deleted
I20260926 13:12:12.099462 16780 ts_tablet_manager.cc:1918] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:12.099530 16780 log.cc:1196] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/95f8b17655d644e784773727827144bc
I20260926 13:12:12.099785 16780 ts_tablet_manager.cc:1939] T 95f8b17655d644e784773727827144bc P 88ddce8e365a4ab3bb950b5d3c023495: Deleting consensus metadata
I20260926 13:12:12.100641 16279 catalog_manager.cc:4841] TS 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135): tablet 95f8b17655d644e784773727827144bc (table test-table [id=57bcab0f768a4a1c940611ecd6eae3ed]) successfully deleted
W20260926 13:12:12.102523 16277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45747 (error 108)
I20260926 13:12:12.102722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 16486
I20260926 13:12:12.104071 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:12.103953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45747 (local address 127.0.0.1:34874)
0926 13:12:12.104004 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:12.104040 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 13:12:12.104231 16277 catalog_manager.cc:4584] TS c423f60803b4469a93d739027ebe5450 (127.0.98.1:45747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95f8b17655d644e784773727827144bc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45747: connect: Connection refused (error 111)
I20260926 13:12:12.107146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 16624
I20260926 13:12:12.111824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 16259
I20260926 13:12:32.117106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40789
--webserver_interface=127.0.98.62
--webserver_port=42261
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:32.197336 16782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:32.197567 16782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:32.197589 16782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:32.198789 16782 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 13:12:32.198865 16782 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 13:12:32.198879 16782 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 13:12:32.198889 16782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:12:32.198902 16782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:32.198918 16782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:32.198975 16782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:12:32.199012 16782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:12:32.200613 16782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:32.200865 16782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:32.201107 16782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:32.207396 16788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:32.207429 16787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:32.207473 16782 server_base.cc:1034] running on GCE node
W20260926 13:12:32.207430 16790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:32.207980 16782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:32.208354 16782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:32.209525 16782 hybrid_clock.cc:648] HybridClock initialized: now 1790428352209499 us; error 44 us; skew 500 ppm
Sep 26 13:12:32 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:32.212631 16782 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:12:32.213176 16782 webserver.cc:460] Webserver started at http://127.0.98.62:42261/ using document root <none> and password file <none>
I20260926 13:12:32.213428 16782 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:32.213479 16782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:32.215095 16782 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:12:32.215801 16797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:32.216007 16782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:32.216099 16782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "3344c043df7e46ef9bbfe07db4a7c943"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "d6b61fb64de8d84cde4cf7ceb89b9bbc"
server_key_iv: "05897514493cd8878fd3b6e324ddc5d6"
server_key_version: "encryptionkey@0"
I20260926 13:12:32.216512 16782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:32.229357 16782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:32.231719 16782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:32.231865 16782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:32.236675 16782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40789
I20260926 13:12:32.236730 16849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40789 every 8 connection(s)
I20260926 13:12:32.237136 16782 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:12:32.238826 16850 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:12:32.239565 16850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Bootstrap starting.
I20260926 13:12:32.243093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 16782
I20260926 13:12:32.243419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45747
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40789
--webserver_port=42721
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:12:32.248956 16850 log.cc:824] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:32.252332 16850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:12:32.252703 16850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Bootstrap complete.
I20260926 13:12:32.256153 16850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260926 13:12:32.256510 16850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3344c043df7e46ef9bbfe07db4a7c943, State: Initialized, Role: FOLLOWER
I20260926 13:12:32.256675 16850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260926 13:12:32.256748 16850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:12:32.256804 16850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:12:32.256862 16850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 2 FOLLOWER]: Advancing to term 3
I20260926 13:12:32.258224 16850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260926 13:12:32.258356 16850 leader_election.cc:302] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3344c043df7e46ef9bbfe07db4a7c943; no voters:
I20260926 13:12:32.258554 16850 leader_election.cc:288] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 13:12:32.258672 16854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 13:12:32.258910 16850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:12:32.258998 16854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [term 3 LEADER]: Becoming Leader. State: Replica: 3344c043df7e46ef9bbfe07db4a7c943, State: Running, Role: LEADER
I20260926 13:12:32.259131 16854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } }
I20260926 13:12:32.259523 16855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3344c043df7e46ef9bbfe07db4a7c943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } } }
I20260926 13:12:32.259550 16854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3344c043df7e46ef9bbfe07db4a7c943. Latest consensus state: current_term: 3 leader_uuid: "3344c043df7e46ef9bbfe07db4a7c943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3344c043df7e46ef9bbfe07db4a7c943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40789 } } }
I20260926 13:12:32.259686 16855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:32.259696 16854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:32.260087 16864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:12:32.261189 16864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:12:32.261327 16864 catalog_manager.cc:1259] Loaded cluster ID: be9185f14b4040dbae12cda1d987e95a
I20260926 13:12:32.261361 16864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:12:32.262241 16864 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:12:32.262428 16864 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Loaded TSK: 0
I20260926 13:12:32.262466 16864 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3344c043df7e46ef9bbfe07db4a7c943: Loaded TSK: 1
F20260926 13:12:32.262696 16864 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff12e9fb9ad google::LogMessage::Fail() at ??:0
@ 0x7ff12e9fe88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff12e9fb4de google::LogMessage::Flush() at ??:0
@ 0x7ff12e9fca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff131168104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff131165d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff1369551cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff13695548e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff136955594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff13691bc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff136925ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff1308c90d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff1308be954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff1317226db start_thread at ??:0
@ 0x7ff12d4f961f clone at ??:0
W20260926 13:12:32.364403 16852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:32.364706 16852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:32.364743 16852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:32.366554 16852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:32.366627 16852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:32.366669 16852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:12:32.368899 16852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42721
--tserver_master_addrs=127.0.98.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:32.369184 16852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:32.369458 16852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:32.375633 16894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:32.375689 16892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:32.375677 16891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:32.376132 16852 server_base.cc:1034] running on GCE node
I20260926 13:12:32.376417 16852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:32.376720 16852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:32.377890 16852 hybrid_clock.cc:648] HybridClock initialized: now 1790428352377880 us; error 32 us; skew 500 ppm
Sep 26 13:12:32 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:32.381197 16852 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:12:32.381829 16852 webserver.cc:460] Webserver started at http://127.0.98.1:42721/ using document root <none> and password file <none>
I20260926 13:12:32.382102 16852 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:32.382175 16852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:32.384087 16852 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:32.384918 16901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:32.385116 16852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:32.385217 16852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "c423f60803b4469a93d739027ebe5450"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "701ab8d8eff39f333cd700d7ef95e70e"
server_key_iv: "cf77ab6ea847bec8790a3388b2dad95f"
server_key_version: "encryptionkey@0"
I20260926 13:12:32.385681 16852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:32.402752 16852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:32.405275 16852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:32.405438 16852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:32.405730 16852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:32.406296 16908 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:12:32.408186 16852 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:12:32.408272 16852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 13:12:32.408310 16852 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:12:32.409466 16852 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:12:32.409518 16852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 13:12:32.409626 16908 tablet_bootstrap.cc:492] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450: Bootstrap starting.
I20260926 13:12:32.417716 16852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45747
I20260926 13:12:32.418242 16852 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:12:32.419292 17015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45747 every 8 connection(s)
I20260926 13:12:32.420958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 16852
I20260926 13:12:32.421352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43641
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40789
--webserver_port=43221
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:12:32.422686 16908 log.cc:824] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:32.425474 16908 tablet_bootstrap.cc:492] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:12:32.425767 16908 tablet_bootstrap.cc:492] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450: Bootstrap complete.
I20260926 13:12:32.426654 16908 ts_tablet_manager.cc:1397] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.006s
I20260926 13:12:32.427847 16908 raft_consensus.cc:348] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:32.428113 16908 raft_consensus.cc:729] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c423f60803b4469a93d739027ebe5450, State: Initialized, Role: FOLLOWER
I20260926 13:12:32.428283 16908 consensus_queue.cc:260] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:32.428632 16908 ts_tablet_manager.cc:1428] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
W20260926 13:12:32.518811 17018 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:12:32.419166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:59611)
0926 13:12:32.419368 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:32.419426 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:32.419938 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:32.518580 (+ 98642us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":116,"threads_started":1}
W20260926 13:12:32.519217 17016 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40789 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40789: Client connection negotiation failed: client connection to 127.0.98.62:40789: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:12:32.519690 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:32.519527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:52347)
0926 13:12:32.519607 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:32.519666 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 13:12:32.520057 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:32.519956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:38089)
0926 13:12:32.520007 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:32.520039 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 13:12:32.520283 17016 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:12:32.535974 17020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:32.536197 17020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:32.536230 17020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:32.537448 17020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:32.537492 17020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:32.537518 17020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:12:32.539076 17020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43221
--tserver_master_addrs=127.0.98.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:32.539302 17020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:32.539501 17020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:32.544888 17026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:32.544934 17029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:32.544934 17027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:32.545447 17020 server_base.cc:1034] running on GCE node
I20260926 13:12:32.545761 17020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:32.546093 17020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:32.547251 17020 hybrid_clock.cc:648] HybridClock initialized: now 1790428352547237 us; error 41 us; skew 500 ppm
Sep 26 13:12:32 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:32.550422 17020 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:12:32.551028 17020 webserver.cc:460] Webserver started at http://127.0.98.2:43221/ using document root <none> and password file <none>
I20260926 13:12:32.551286 17020 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:32.551343 17020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:32.552793 17020 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:32.553514 17036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:32.553735 17020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:32.553830 17020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "b8381811d2e449ce8d9ecb43b1449369"
format_stamp: "Formatted at 2026-09-26 13:11:50 on dist-test-slave-2dw5"
server_key: "ed7cb033f9a7f6ef2fc459c7b9c4d97b"
server_key_iv: "8d6fdba344917fdb920d76aa84337e6d"
server_key_version: "encryptionkey@0"
I20260926 13:12:32.554227 17020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:32.566740 17020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:32.569101 17020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:32.569242 17020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:32.569483 17020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:32.569803 17020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:32.569839 17020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:32.569867 17020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:32.569883 17020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:32.576495 17020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43641
I20260926 13:12:32.576618 17149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43641 every 8 connection(s)
I20260926 13:12:32.577049 17020 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:12:32.578436 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:32.577956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:53031)
0926 13:12:32.578158 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:32.578254 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260926 13:12:32.578864 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 17020
W20260926 13:12:32.578974 17150 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40789 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40789: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
I20260926 13:12:32.579162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35135
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40789
--webserver_port=46281
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:12:32.579325 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:32.579227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:60237)
0926 13:12:32.579278 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:32.579300 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 13:12:32.579588 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:32.579510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:41451)
0926 13:12:32.579547 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:32.579569 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 13:12:32.579731 17150 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:12:32.663941 17153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:32.664201 17153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:32.664227 17153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:32.665458 17153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:32.665517 17153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:32.665545 17153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:12:32.667124 17153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46281
--tserver_master_addrs=127.0.98.62:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:32.667375 17153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:32.667608 17153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:12:32.673101 17153 server_base.cc:1034] running on GCE node
W20260926 13:12:32.673041 17158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:32.673041 17159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:32.673064 17161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:32.673589 17153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:32.673875 17153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:32.675092 17153 hybrid_clock.cc:648] HybridClock initialized: now 1790428352675069 us; error 34 us; skew 500 ppm
Sep 26 13:12:32 dist-test-slave-2dw5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:32.677940 17153 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:12:32.678504 17153 webserver.cc:460] Webserver started at http://127.0.98.3:46281/ using document root <none> and password file <none>
I20260926 13:12:32.678754 17153 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:32.678802 17153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:32.680454 17153 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:32.681267 17168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:32.681520 17153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:32.681619 17153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "88ddce8e365a4ab3bb950b5d3c023495"
format_stamp: "Formatted at 2026-09-26 13:11:51 on dist-test-slave-2dw5"
server_key: "43ae988bf1c3a9ffc82c75328e6457d7"
server_key_iv: "e648003bc547c8570e12e8df9b3d857b"
server_key_version: "encryptionkey@0"
I20260926 13:12:32.682078 17153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:32.703977 17153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:32.706468 17153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:32.706609 17153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:32.706913 17153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:32.707266 17153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:32.707301 17153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:32.707326 17153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:32.707350 17153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:32.714289 17153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35135
I20260926 13:12:32.714381 17281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35135 every 8 connection(s)
I20260926 13:12:32.714741 17153 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:12:32.714982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 17153
I20260926 13:12:32.715986 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:32.715544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:52491)
0926 13:12:32.715747 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:32.715825 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
W20260926 13:12:32.716414 17282 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40789 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40789: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
I20260926 13:12:32.716753 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:32.716640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:34913)
0926 13:12:32.716696 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:32.716730 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 13:12:32.717039 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:32.716974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:40355)
0926 13:12:32.717004 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:32.717027 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260926 13:12:32.717142 17282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:12:33.521100 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:33.520761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:59429)
0926 13:12:33.520935 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:33.521001 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260926 13:12:33.580469 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:33.580175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:41463)
0926 13:12:33.580323 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:33.580375 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260926 13:12:33.717743 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:33.717437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:47313)
0926 13:12:33.717608 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:33.717669 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260926 13:12:33.861651 17289 raft_consensus.cc:482] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:33.861812 17289 raft_consensus.cc:504] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:33.862165 17289 leader_election.cc:288] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135), b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641)
Sep 26 13:12:33 dist-test-slave-2dw5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 26 13:12:33 dist-test-slave-2dw5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260926 13:12:33.879691 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:33.862639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58735 (local address 127.0.98.3:35135)
0926 13:12:33.862829 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:33.862870 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:33.862906 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:12:33.863456 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:33.863459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:33.863511 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:33.863645 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:33.864609 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:33.865630 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:33.866867 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:33.867314 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:33.867426 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:33.875871 (+ 8445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:33.875875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:33.875876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:33.875889 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:33.877876 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:33.878349 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:33.878351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:33.878352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:33.878446 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:33.878755 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:33.878756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:33.878756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:33.879059 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:33.879136 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:33.879427 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 13:12:33.879569 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260926 13:12:33.879691 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:33.862437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35135 (local address 127.0.98.1:58735)
0926 13:12:33.862523 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:33.862538 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:33.862637 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:33.863725 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:33.863730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:33.864183 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:33.864463 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:33.864473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:33.865766 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:33.865768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:33.866704 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:33.866708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:33.867401 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:33.867402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:33.867535 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:33.873337 (+ 5802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:33.873368 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:33.875739 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:33.877994 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:33.878002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:33.878004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:33.878263 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:33.878544 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:33.878545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:33.878545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:33.878687 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:33.879130 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:33.879140 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:33.879270 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 13:12:33.879378 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 13:12:33.879594 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":71}
I20260926 13:12:33.879832 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:33.862437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43641 (local address 127.0.98.1:55797)
0926 13:12:33.862736 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:33.862746 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:33.862797 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:33.863739 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:33.863741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:33.864173 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:33.864462 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:33.864470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:33.865678 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:33.865679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:33.866702 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:33.866705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:33.867396 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:33.867398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:33.867539 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:33.873339 (+ 5800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:33.873367 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:33.875406 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:33.878350 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:33.878350 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:33.878351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:33.878481 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:33.878794 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:33.878795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:33.878796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:33.878881 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:33.879298 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:33.879299 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:33.879380 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 13:12:33.879439 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:12:33.879685 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":376,"thread_start_us":71,"threads_started":1}
I20260926 13:12:33.879819 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:33.862582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55797 (local address 127.0.98.2:43641)
0926 13:12:33.862827 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:33.862874 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:33.862920 (+ 46us) server_negotiation.cc:407] Connection header received
0926 13:12:33.863507 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:33.863510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:33.863562 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:33.863718 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:33.864581 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:33.865577 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:33.866786 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:33.867307 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:33.867432 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:33.875601 (+ 8169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:33.875608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:33.875608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:33.875624 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:33.877876 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:33.878592 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:33.878596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:33.878597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:33.878726 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:33.878940 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:33.878941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:33.878942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:33.879227 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:33.879306 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:33.879530 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 13:12:33.879683 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
W20260926 13:12:33.880785 16904 leader_election.cc:341] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641): Not found: Tablet not found: 95f8b17655d644e784773727827144bc
W20260926 13:12:33.880927 16905 leader_election.cc:341] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135): Not found: Tablet not found: 95f8b17655d644e784773727827144bc
I20260926 13:12:33.881004 16905 leader_election.cc:302] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c423f60803b4469a93d739027ebe5450; no voters: 88ddce8e365a4ab3bb950b5d3c023495, b8381811d2e449ce8d9ecb43b1449369
I20260926 13:12:33.881172 17289 raft_consensus.cc:2726] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:12:34.522189 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:34.521868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:56897)
0926 13:12:34.522078 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:34.522140 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
I20260926 13:12:34.581271 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:34.580982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:33781)
0926 13:12:34.581145 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:34.581211 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260926 13:12:34.718654 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:34.718268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:36147)
0926 13:12:34.718542 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:34.718603 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":174,"threads_started":1}
I20260926 13:12:35.523226 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:35.522941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:60895)
0926 13:12:35.523118 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:35.523178 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
I20260926 13:12:35.542820 17297 raft_consensus.cc:482] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:35.543004 17297 raft_consensus.cc:504] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:35.543195 17297 leader_election.cc:288] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135), b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641)
W20260926 13:12:35.543675 16905 leader_election.cc:341] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135): Not found: Tablet not found: 95f8b17655d644e784773727827144bc
W20260926 13:12:35.543774 16904 leader_election.cc:341] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641): Not found: Tablet not found: 95f8b17655d644e784773727827144bc
I20260926 13:12:35.543879 16904 leader_election.cc:302] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c423f60803b4469a93d739027ebe5450; no voters: 88ddce8e365a4ab3bb950b5d3c023495, b8381811d2e449ce8d9ecb43b1449369
I20260926 13:12:35.544013 17297 raft_consensus.cc:2726] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:12:35.582093 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:35.581814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:56553)
0926 13:12:35.581982 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:35.582041 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260926 13:12:35.719508 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:35.719208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:47821)
0926 13:12:35.719371 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:35.719465 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260926 13:12:36.524327 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:36.523962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:53199)
0926 13:12:36.524185 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:36.524257 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
I20260926 13:12:36.583014 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:36.582650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:48615)
0926 13:12:36.582834 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:36.582953 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
I20260926 13:12:36.720361 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:36.720062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:56257)
0926 13:12:36.720269 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:36.720312 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":106,"threads_started":1}
I20260926 13:12:37.525384 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:37.525026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:47667)
0926 13:12:37.525253 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:37.525330 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":105,"threads_started":1}
I20260926 13:12:37.583848 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:37.583614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:50135)
0926 13:12:37.583751 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:37.583781 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260926 13:12:37.721221 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:37.720930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:56653)
0926 13:12:37.721109 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:37.721165 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":71,"threads_started":1}
I20260926 13:12:38.291271 17306 raft_consensus.cc:482] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:38.291391 17306 raft_consensus.cc:504] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:38.291584 17306 leader_election.cc:288] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135), b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641)
W20260926 13:12:38.292035 16905 leader_election.cc:341] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135): Not found: Tablet not found: 95f8b17655d644e784773727827144bc
W20260926 13:12:38.292142 16904 leader_election.cc:341] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641): Not found: Tablet not found: 95f8b17655d644e784773727827144bc
I20260926 13:12:38.292210 16904 leader_election.cc:302] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c423f60803b4469a93d739027ebe5450; no voters: 88ddce8e365a4ab3bb950b5d3c023495, b8381811d2e449ce8d9ecb43b1449369
I20260926 13:12:38.292346 17306 raft_consensus.cc:2726] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:12:38.526324 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:38.526012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:37317)
0926 13:12:38.526201 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:38.526258 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260926 13:12:38.584652 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:38.584382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:46619)
0926 13:12:38.584548 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:38.584603 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260926 13:12:38.722187 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:38.721876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:59215)
0926 13:12:38.722068 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:38.722134 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260926 13:12:39.527182 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:39.526925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:58755)
0926 13:12:39.527082 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:39.527132 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260926 13:12:39.585496 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:39.585165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:58565)
0926 13:12:39.585378 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:39.585425 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":131,"threads_started":1}
I20260926 13:12:39.723071 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:39.722759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:45075)
0926 13:12:39.722964 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:39.723020 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":133,"threads_started":1}
I20260926 13:12:40.528002 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:40.527729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:49531)
0926 13:12:40.527911 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:40.527955 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":103,"threads_started":1}
I20260926 13:12:40.586179 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:40.585957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:33999)
0926 13:12:40.586094 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:40.586132 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260926 13:12:40.723755 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:40.723547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:53127)
0926 13:12:40.723667 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:40.723708 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260926 13:12:41.528791 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:41.528525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:56623)
0926 13:12:41.528686 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:41.528742 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260926 13:12:41.587046 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:41.586693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:41075)
0926 13:12:41.586917 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:41.586981 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":70,"threads_started":1}
I20260926 13:12:41.724697 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:41.724350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:57823)
0926 13:12:41.724570 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:41.724637 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260926 13:12:42.529736 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:42.529461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:48955)
0926 13:12:42.529630 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:42.529687 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260926 13:12:42.588073 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:42.587720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:39853)
0926 13:12:42.587896 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:42.588019 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260926 13:12:42.653544 17321 raft_consensus.cc:482] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:42.653699 17321 raft_consensus.cc:504] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:42.653934 17321 leader_election.cc:288] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135), b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641)
W20260926 13:12:42.654559 16904 leader_election.cc:341] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641): Not found: Tablet not found: 95f8b17655d644e784773727827144bc
W20260926 13:12:42.654690 16905 leader_election.cc:341] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135): Not found: Tablet not found: 95f8b17655d644e784773727827144bc
I20260926 13:12:42.654726 16905 leader_election.cc:302] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c423f60803b4469a93d739027ebe5450; no voters: 88ddce8e365a4ab3bb950b5d3c023495, b8381811d2e449ce8d9ecb43b1449369
I20260926 13:12:42.654886 17321 raft_consensus.cc:2726] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:12:42.725656 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:42.725352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:44877)
0926 13:12:42.725518 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:42.725570 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260926 13:12:43.530634 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:43.530287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:36835)
0926 13:12:43.530529 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:43.530586 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
I20260926 13:12:43.588802 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:43.588584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:51763)
0926 13:12:43.588715 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:43.588755 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260926 13:12:43.726514 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:43.726192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:34305)
0926 13:12:43.726406 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:43.726451 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260926 13:12:44.531440 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:44.531134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:38265)
0926 13:12:44.531310 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:44.531376 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260926 13:12:44.589723 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:44.589427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:41275)
0926 13:12:44.589599 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:44.589660 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":177,"threads_started":1}
I20260926 13:12:44.727412 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:44.727092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:37791)
0926 13:12:44.727306 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:44.727367 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
I20260926 13:12:45.532418 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:45.532087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:54715)
0926 13:12:45.532290 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:45.532355 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20260926 13:12:45.590684 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:45.590349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:53895)
0926 13:12:45.590572 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:45.590636 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":112,"threads_started":1}
I20260926 13:12:45.728513 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:45.728154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:41691)
0926 13:12:45.728386 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:45.728449 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":120,"threads_started":1}
I20260926 13:12:46.180766 17332 raft_consensus.cc:482] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:46.180873 17332 raft_consensus.cc:504] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88ddce8e365a4ab3bb950b5d3c023495" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35135 } } peers { permanent_uuid: "b8381811d2e449ce8d9ecb43b1449369" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43641 } } peers { permanent_uuid: "c423f60803b4469a93d739027ebe5450" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45747 } }
I20260926 13:12:46.181052 17332 leader_election.cc:288] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135), b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641)
W20260926 13:12:46.181654 16904 leader_election.cc:341] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8381811d2e449ce8d9ecb43b1449369 (127.0.98.2:43641): Not found: Tablet not found: 95f8b17655d644e784773727827144bc
W20260926 13:12:46.181783 16905 leader_election.cc:341] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 88ddce8e365a4ab3bb950b5d3c023495 (127.0.98.3:35135): Not found: Tablet not found: 95f8b17655d644e784773727827144bc
I20260926 13:12:46.181860 16905 leader_election.cc:302] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c423f60803b4469a93d739027ebe5450; no voters: 88ddce8e365a4ab3bb950b5d3c023495, b8381811d2e449ce8d9ecb43b1449369
I20260926 13:12:46.181949 17332 raft_consensus.cc:2726] T 95f8b17655d644e784773727827144bc P c423f60803b4469a93d739027ebe5450 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:12:46.533517 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:46.533189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:59221)
0926 13:12:46.533399 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:46.533466 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260926 13:12:46.591620 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:46.591351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:39335)
0926 13:12:46.591512 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:46.591569 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260926 13:12:46.729384 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:46.729098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.3:46305)
0926 13:12:46.729259 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:46.729320 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260926 13:12:47.534497 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:47.534251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.1:53343)
0926 13:12:47.534399 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:47.534444 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260926 13:12:47.592459 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:47.592262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40789 (local address 127.0.98.2:39211)
0926 13:12:47.592381 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:47.592411 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260926 13:12:47.718685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 16852
I20260926 13:12:47.723193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 17020
I20260926 13:12:47.727530 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 17153
2026-09-26T13:12:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58265 ms)
[----------] 30 tests from SecurityITest (231618 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:12:47 dist-test-slave-2dw5 krb5kdc[17340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:12:47 dist-test-slave-2dw5 krb5kdc[17340](info): set up 2 sockets
Sep 26 13:12:47 dist-test-slave-2dw5 krb5kdc[17340](info): commencing operation
krb5kdc: starting...
W20260926 13:12:48.706406 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:12:48 dist-test-slave-2dw5 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:12:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:12:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:12:48.831710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41439
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:48.916251 17353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:48.916527 17353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:48.916553 17353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:48.917855 17353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:12:48.917909 17353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:48.917923 17353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:48.917935 17353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:12:48.917946 17353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:12:48.919553 17353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:48.919786 17353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:48.919988 17353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:48.925302 17361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:48.925325 17359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:48.925278 17358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:48.925324 17353 server_base.cc:1034] running on GCE node
I20260926 13:12:48.925670 17353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:48.925959 17353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:48.927132 17353 hybrid_clock.cc:648] HybridClock initialized: now 1790428368927113 us; error 35 us; skew 500 ppm
Sep 26 13:12:48 dist-test-slave-2dw5 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:48.929858 17353 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:12:48.930400 17353 webserver.cc:460] Webserver started at http://127.0.98.62:45655/ using document root <none> and password file <none>
I20260926 13:12:48.930612 17353 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:48.930661 17353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:48.930800 17353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:48.931931 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "19f2668ace1f4fe5ae29b199c64d2587"
format_stamp: "Formatted at 2026-09-26 13:12:48 on dist-test-slave-2dw5"
server_key: "b822df99da2a6785badb5c860e21ef8b"
server_key_iv: "826233be12b2a4f472f60a76286c0185"
server_key_version: "encryptionkey@0"
I20260926 13:12:48.932299 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "19f2668ace1f4fe5ae29b199c64d2587"
format_stamp: "Formatted at 2026-09-26 13:12:48 on dist-test-slave-2dw5"
server_key: "b822df99da2a6785badb5c860e21ef8b"
server_key_iv: "826233be12b2a4f472f60a76286c0185"
server_key_version: "encryptionkey@0"
I20260926 13:12:48.933619 17353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:12:48.934397 17368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:48.934578 17353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:48.934674 17353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "19f2668ace1f4fe5ae29b199c64d2587"
format_stamp: "Formatted at 2026-09-26 13:12:48 on dist-test-slave-2dw5"
server_key: "b822df99da2a6785badb5c860e21ef8b"
server_key_iv: "826233be12b2a4f472f60a76286c0185"
server_key_version: "encryptionkey@0"
I20260926 13:12:48.934751 17353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:48.950100 17353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:48.952241 17353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:48.952387 17353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:48.956368 17353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41439
I20260926 13:12:48.956415 17420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41439 every 8 connection(s)
I20260926 13:12:48.956743 17353 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:12:48.957023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 17353
I20260926 13:12:48.957119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:12:48.957321 392 external_mini_cluster.cc:1428] Setting key 9208f5b3f0004daf90f176ac240bc5a1
I20260926 13:12:48.957739 17421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:48.961124 17421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587: Bootstrap starting.
I20260926 13:12:48.961768 17421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:48.962097 17421 log.cc:824] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:48.962953 17421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587: No bootstrap required, opened a new log
Sep 26 13:12:48 dist-test-slave-2dw5 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:48.965018 17421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f2668ace1f4fe5ae29b199c64d2587" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41439 } }
I20260926 13:12:48.965181 17421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:48.965219 17421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f2668ace1f4fe5ae29b199c64d2587, State: Initialized, Role: FOLLOWER
I20260926 13:12:48.965353 17421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f2668ace1f4fe5ae29b199c64d2587" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41439 } }
I20260926 13:12:48.965432 17421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:12:48.965468 17421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:12:48.965518 17421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:48.966135 17421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f2668ace1f4fe5ae29b199c64d2587" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41439 } }
I20260926 13:12:48.966238 17421 leader_election.cc:302] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19f2668ace1f4fe5ae29b199c64d2587; no voters:
I20260926 13:12:48.966413 17421 leader_election.cc:288] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:12:48.966471 17426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:48.966622 17426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [term 1 LEADER]: Becoming Leader. State: Replica: 19f2668ace1f4fe5ae29b199c64d2587, State: Running, Role: LEADER
I20260926 13:12:48.966684 17421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:12:48.966722 17426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f2668ace1f4fe5ae29b199c64d2587" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41439 } }
I20260926 13:12:48.967106 17426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19f2668ace1f4fe5ae29b199c64d2587. Latest consensus state: current_term: 1 leader_uuid: "19f2668ace1f4fe5ae29b199c64d2587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f2668ace1f4fe5ae29b199c64d2587" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41439 } } }
I20260926 13:12:48.967192 17426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:48.967113 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19f2668ace1f4fe5ae29b199c64d2587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f2668ace1f4fe5ae29b199c64d2587" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41439 } } }
I20260926 13:12:48.967382 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:48.967769 17432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:12:48.968364 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:48.957963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.0.98.62:41439)
0926 13:12:48.958158 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:48.958167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:48.958199 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:12:48.958857 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:48.958867 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:48.958934 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:48.959083 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:48.959638 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:48.960987 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:48.961838 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:48.962428 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:48.962530 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:48.964799 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:48.964804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:48.964804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:48.964815 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:48.966455 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:48.966884 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:48.966887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:48.966887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:48.966983 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:48.967192 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:48.967192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:48.967193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:48.967533 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:48.967678 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:48.968028 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 13:12:48.968167 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260926 13:12:48.968412 17432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:12:48.970113 17432 catalog_manager.cc:1347] Generated new cluster ID: 8e5022afda49476eb7331b19bc7ecd31
I20260926 13:12:48.970160 17432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:12:48.978650 17432 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:12:48.979455 17432 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:12:48.984733 17432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587: Generated new TSK 0
I20260926 13:12:48.984939 17432 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:12:49.023386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41439
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:49.125797 17447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:49.126030 17447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:49.126065 17447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:49.127347 17447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:49.127394 17447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:49.127434 17447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:12:49.129074 17447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:49.129328 17447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:49.129539 17447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:49.134881 17455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:49.134907 17447 server_base.cc:1034] running on GCE node
W20260926 13:12:49.134862 17452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:49.134862 17453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:49.135380 17447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:49.135644 17447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:49.136826 17447 hybrid_clock.cc:648] HybridClock initialized: now 1790428369136792 us; error 50 us; skew 500 ppm
Sep 26 13:12:49 dist-test-slave-2dw5 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:49.139748 17447 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:12:49.140241 17447 webserver.cc:460] Webserver started at http://127.0.98.1:34593/ using document root <none> and password file <none>
I20260926 13:12:49.140458 17447 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:49.140515 17447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:49.140640 17447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:49.141714 17447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece"
format_stamp: "Formatted at 2026-09-26 13:12:49 on dist-test-slave-2dw5"
server_key: "a5471f7005232705adeaffcbc0759cd8"
server_key_iv: "bdefd6c794fab1719fd145cc11dde901"
server_key_version: "encryptionkey@0"
I20260926 13:12:49.142045 17447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece"
format_stamp: "Formatted at 2026-09-26 13:12:49 on dist-test-slave-2dw5"
server_key: "a5471f7005232705adeaffcbc0759cd8"
server_key_iv: "bdefd6c794fab1719fd145cc11dde901"
server_key_version: "encryptionkey@0"
I20260926 13:12:49.143455 17447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:12:49.144228 17462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:49.144434 17447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:12:49.144527 17447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece"
format_stamp: "Formatted at 2026-09-26 13:12:49 on dist-test-slave-2dw5"
server_key: "a5471f7005232705adeaffcbc0759cd8"
server_key_iv: "bdefd6c794fab1719fd145cc11dde901"
server_key_version: "encryptionkey@0"
I20260926 13:12:49.144605 17447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:49.162556 17447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:49.164760 17447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:49.164902 17447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:49.165120 17447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:49.165416 17447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:49.165453 17447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:49.165482 17447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:49.165503 17447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:49.171306 17447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36161
I20260926 13:12:49.171373 17575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36161 every 8 connection(s)
I20260926 13:12:49.171669 17447 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:12:49 dist-test-slave-2dw5 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:49.179266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 17447
I20260926 13:12:49.179374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:12:49.179569 392 external_mini_cluster.cc:1428] Setting key 8f6d355a2f090d2f87c0d5e1ea5fb6f2
I20260926 13:12:49.181507 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.172493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42219 (local address 127.0.98.62:41439)
0926 13:12:49.172583 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.172585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.172783 (+ 198us) server_negotiation.cc:407] Connection header received
0926 13:12:49.173801 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.173802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.173816 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.173868 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:49.174533 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.175068 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.176238 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.176451 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.176513 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.178717 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:49.178719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:49.178719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:49.178728 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:49.180106 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:49.180536 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:49.180537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:49.180538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:49.180599 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:49.180833 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:49.180834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:49.180834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:49.180952 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:49.181009 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.181303 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.181411 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:12:49.181653 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.172445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41439 (local address 127.0.98.1:42219)
0926 13:12:49.172640 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:49.172693 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:49.173639 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:49.173941 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:49.173945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:49.174216 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:49.174433 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.174440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.175149 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.175151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.176033 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.176035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.176534 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.176535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.176662 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.177175 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:49.177183 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:49.178613 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:49.180197 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:49.180207 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:49.180210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:49.180462 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:49.180656 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:49.180657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:49.180657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:49.180778 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:49.181008 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:49.181015 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:49.181162 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 13:12:49.181261 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 13:12:49.181431 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260926 13:12:49.182077 17576 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41439
I20260926 13:12:49.182175 17576 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:49.182355 17576 heartbeater.cc:511] Master 127.0.98.62:41439 requested a full tablet report, sending...
I20260926 13:12:49.182915 17385 ts_manager.cc:194] Registered new tserver with Master: bd1a82eb5ce543a4bbf2851ace6d5ece (127.0.98.1:36161)
I20260926 13:12:49.183560 17385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42219
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:12:49.206950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41439
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:49.286782 17581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:49.287052 17581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:49.287086 17581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:49.288275 17581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:49.288319 17581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:49.288357 17581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:12:49.289908 17581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:49.290164 17581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:49.290374 17581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:12:49.295693 17581 server_base.cc:1034] running on GCE node
W20260926 13:12:49.295595 17589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:49.295603 17586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:49.295611 17587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:49.296039 17581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:49.296289 17581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:49.297431 17581 hybrid_clock.cc:648] HybridClock initialized: now 1790428369297414 us; error 34 us; skew 500 ppm
Sep 26 13:12:49 dist-test-slave-2dw5 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:49.308759 17581 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:12:49.309298 17581 webserver.cc:460] Webserver started at http://127.0.98.2:39417/ using document root <none> and password file <none>
I20260926 13:12:49.309530 17581 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:49.309581 17581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:49.309723 17581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:49.310801 17581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e"
format_stamp: "Formatted at 2026-09-26 13:12:49 on dist-test-slave-2dw5"
server_key: "0f2b6f21ffbbaf2a64ddc75683387c00"
server_key_iv: "87dddd2612b6c07b3b0183da04746cda"
server_key_version: "encryptionkey@0"
I20260926 13:12:49.311194 17581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e"
format_stamp: "Formatted at 2026-09-26 13:12:49 on dist-test-slave-2dw5"
server_key: "0f2b6f21ffbbaf2a64ddc75683387c00"
server_key_iv: "87dddd2612b6c07b3b0183da04746cda"
server_key_version: "encryptionkey@0"
I20260926 13:12:49.312533 17581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:49.313269 17596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:49.313406 17581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:12:49.313490 17581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e"
format_stamp: "Formatted at 2026-09-26 13:12:49 on dist-test-slave-2dw5"
server_key: "0f2b6f21ffbbaf2a64ddc75683387c00"
server_key_iv: "87dddd2612b6c07b3b0183da04746cda"
server_key_version: "encryptionkey@0"
I20260926 13:12:49.313558 17581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:49.321210 17581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:49.323287 17581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:49.323423 17581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:49.323719 17581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:49.324014 17581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:49.324051 17581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:49.324080 17581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:49.324097 17581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:49.329937 17581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46037
I20260926 13:12:49.329977 17709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46037 every 8 connection(s)
I20260926 13:12:49.330318 17581 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:12:49.331673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 17581
I20260926 13:12:49.331755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:12:49.331929 392 external_mini_cluster.cc:1428] Setting key 2501450bd59185004ef7ed7ca912562a
Sep 26 13:12:49 dist-test-slave-2dw5 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 13:12:49.340999 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.331044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58557 (local address 127.0.98.62:41439)
0926 13:12:49.331099 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.331101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.331299 (+ 198us) server_negotiation.cc:407] Connection header received
0926 13:12:49.332375 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.332377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.332388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.332436 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:49.333099 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.333595 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.334562 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.334763 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.334815 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.337401 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:49.337403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:49.337403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:49.337410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:49.339014 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:49.339467 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:49.339468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:49.339469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:49.339543 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:49.339838 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:49.339839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:49.339839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:49.340362 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:49.340430 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.340727 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.340907 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 13:12:49.341109 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.331014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41439 (local address 127.0.98.2:58557)
0926 13:12:49.331182 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:49.331222 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:49.332285 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:49.332505 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:49.332509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:49.332819 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:49.333012 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.333017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.333680 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.333681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.334484 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.334486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.334828 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.334829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.334976 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.335519 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:49.335530 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:49.337304 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:49.339103 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:49.339112 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:49.339114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:49.339396 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:49.339616 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:49.339617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:49.339617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:49.339769 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:49.340429 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:49.340437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:49.340606 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 13:12:49.340701 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 13:12:49.340859 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260926 13:12:49.341722 17710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41439
I20260926 13:12:49.341821 17710 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:49.342016 17710 heartbeater.cc:511] Master 127.0.98.62:41439 requested a full tablet report, sending...
I20260926 13:12:49.342453 17385 ts_manager.cc:194] Registered new tserver with Master: 32b378aa6b114b2ea0e03eadc3cd5e3e (127.0.98.2:46037)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 13:12:49.342984 17385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:12:49.358294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41439
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:49.452042 17715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:49.452360 17715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:49.452498 17715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:49.454097 17715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:49.454159 17715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:49.454192 17715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:12:49.455947 17715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:49.456178 17715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:49.456379 17715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:49.461612 17721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:49.461491 17723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:49.461508 17715 server_base.cc:1034] running on GCE node
W20260926 13:12:49.461508 17720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:49.462000 17715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:49.462213 17715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:49.463392 17715 hybrid_clock.cc:648] HybridClock initialized: now 1790428369463378 us; error 28 us; skew 500 ppm
Sep 26 13:12:49 dist-test-slave-2dw5 krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:49.465875 17715 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:12:49.466293 17715 webserver.cc:460] Webserver started at http://127.0.98.3:46173/ using document root <none> and password file <none>
I20260926 13:12:49.466539 17715 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:49.466624 17715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:49.466755 17715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:49.468057 17715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "b5354496a047489ea8d0b870b5c012db"
format_stamp: "Formatted at 2026-09-26 13:12:49 on dist-test-slave-2dw5"
server_key: "167469de364c2779f005a37138483bda"
server_key_iv: "c1039652cd068baabf70057ce435a995"
server_key_version: "encryptionkey@0"
I20260926 13:12:49.468405 17715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b5354496a047489ea8d0b870b5c012db"
format_stamp: "Formatted at 2026-09-26 13:12:49 on dist-test-slave-2dw5"
server_key: "167469de364c2779f005a37138483bda"
server_key_iv: "c1039652cd068baabf70057ce435a995"
server_key_version: "encryptionkey@0"
I20260926 13:12:49.469674 17715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:12:49.470320 17730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:49.470456 17715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:12:49.470530 17715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "b5354496a047489ea8d0b870b5c012db"
format_stamp: "Formatted at 2026-09-26 13:12:49 on dist-test-slave-2dw5"
server_key: "167469de364c2779f005a37138483bda"
server_key_iv: "c1039652cd068baabf70057ce435a995"
server_key_version: "encryptionkey@0"
I20260926 13:12:49.470602 17715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:49.489372 17715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:49.491603 17715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:49.491779 17715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:49.492023 17715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:49.492336 17715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:49.492376 17715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:49.492410 17715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:49.492452 17715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:49.498909 17715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44973
I20260926 13:12:49.498953 17843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44973 every 8 connection(s)
I20260926 13:12:49.499280 17715 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:12:49.503120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 17715
I20260926 13:12:49.503224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:12:49.503401 392 external_mini_cluster.cc:1428] Setting key 3c5e43f41c660d53da2f895b126211f0
Sep 26 13:12:49 dist-test-slave-2dw5 krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:49.508723 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.500052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53763 (local address 127.0.98.62:41439)
0926 13:12:49.500119 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.500122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.500271 (+ 149us) server_negotiation.cc:407] Connection header received
0926 13:12:49.501223 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.501225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.501236 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.501297 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:49.501921 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.502434 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.503727 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.503988 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.504057 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.506123 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:49.506125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:49.506125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:49.506133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:49.507339 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:49.507760 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:49.507761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:49.507761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:49.507809 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:49.508023 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:49.508023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:49.508023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:49.508130 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:49.508176 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.508595 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.508656 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:12:49.508940 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.499997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41439 (local address 127.0.98.3:53763)
0926 13:12:49.500153 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:49.500190 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:49.501096 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:49.501361 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:49.501365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:49.501627 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:49.501823 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.501827 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.502527 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.502528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.503665 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.503668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.504067 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.504068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.504198 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.504726 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:49.504734 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:49.506038 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:49.507412 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:49.507420 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:49.507422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:49.507699 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:49.507869 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:49.507870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:49.507870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:49.507978 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:49.508211 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:49.508218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:49.508353 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 13:12:49.508578 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 13:12:49.508725 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260926 13:12:49.509511 17844 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41439
I20260926 13:12:49.509618 17844 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:49.509800 17844 heartbeater.cc:511] Master 127.0.98.62:41439 requested a full tablet report, sending...
I20260926 13:12:49.510181 17385 ts_manager.cc:194] Registered new tserver with Master: b5354496a047489ea8d0b870b5c012db (127.0.98.3:44973)
I20260926 13:12:49.510520 17385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53763
I20260926 13:12:49.515290 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:12:49.522511 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.516657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.0.98.62:41439)
0926 13:12:49.516705 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.516707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.516859 (+ 152us) server_negotiation.cc:407] Connection header received
0926 13:12:49.516885 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.516886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.516897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.516954 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:49.517556 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.518079 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.518961 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.519166 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.519220 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.520309 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:49.520310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:49.520311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:49.520317 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:49.521565 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:49.521884 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:49.521885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:49.521885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:49.521935 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:49.522082 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:49.522082 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:49.522083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:49.522223 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:49.522296 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.522379 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.522443 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:12:49.523998 17385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:12:49.525506 17385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:12:49.531934 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.527825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46037 (local address 127.0.0.1:35348)
0926 13:12:49.527959 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:49.527974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:49.528036 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:49.528605 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:49.528606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:49.528607 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:49.528647 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.528648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.529605 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.529606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.530700 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.530702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.531618 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.531619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.531703 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.531717 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 13:12:49.531792 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:12:49.531843 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":39,"thread_start_us":86,"threads_started":1}
I20260926 13:12:49.531944 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.528277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.0.98.2:46037)
0926 13:12:49.528443 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.528447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.528458 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:12:49.528501 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.528503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.528518 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.528609 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:49.528689 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.529501 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.530812 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.531530 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.531606 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.531669 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.531817 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.531861 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260926 13:12:49.532277 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.527810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36161 (local address 127.0.0.1:46564)
0926 13:12:49.527931 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:49.527974 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:49.528036 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:49.528425 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:49.528429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:49.528431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:49.528508 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.528511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.529559 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.529560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.530993 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.530996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.531926 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.531927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.532119 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.532124 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:12:49.532170 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 13:12:49.532211 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":41,"thread_start_us":57,"threads_started":1}
I20260926 13:12:49.532567 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.528097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.0.98.1:36161)
0926 13:12:49.528271 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.528276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.528287 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:12:49.528328 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.528330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.528345 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.528418 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:49.528577 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.529457 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.531082 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.531769 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.531830 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.531880 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.532448 (+ 568us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.532495 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260926 13:12:49.532781 17644 tablet_service.cc:1467] Processing CreateTablet for tablet 97278d00f6bf43b1abea2175862367b3 (DEFAULT_TABLE table=test-table [id=f03f00fd21974f53934abf1e5ae6a91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:49.533157 17644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97278d00f6bf43b1abea2175862367b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:49.533183 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.528004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44973 (local address 127.0.0.1:41044)
0926 13:12:49.529643 (+ 1639us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:49.529654 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:49.529709 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:49.529878 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:49.529879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:49.529881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:49.529936 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.529937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.531178 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.531178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.532062 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.532065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.532976 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.532977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.533035 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.533040 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:12:49.533083 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 13:12:49.533118 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1611,"thread_start_us":66,"threads_started":1}
I20260926 13:12:49.533170 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 97278d00f6bf43b1abea2175862367b3 (DEFAULT_TABLE table=test-table [id=f03f00fd21974f53934abf1e5ae6a91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:49.533443 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.528213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.0.98.3:44973)
0926 13:12:49.528512 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.528516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.529722 (+ 1206us) server_negotiation.cc:407] Connection header received
0926 13:12:49.529774 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.529776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.529793 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.529871 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:49.529998 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.531065 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.532145 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.532824 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.532888 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.532944 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.533317 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.533360 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260926 13:12:49.533458 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97278d00f6bf43b1abea2175862367b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:49.533986 17778 tablet_service.cc:1467] Processing CreateTablet for tablet 97278d00f6bf43b1abea2175862367b3 (DEFAULT_TABLE table=test-table [id=f03f00fd21974f53934abf1e5ae6a91c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:49.534256 17778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97278d00f6bf43b1abea2175862367b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:49.536621 17862 tablet_bootstrap.cc:492] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e: Bootstrap starting.
I20260926 13:12:49.536933 17861 tablet_bootstrap.cc:492] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece: Bootstrap starting.
I20260926 13:12:49.536924 17863 tablet_bootstrap.cc:492] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db: Bootstrap starting.
I20260926 13:12:49.537506 17863 tablet_bootstrap.cc:654] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:49.537509 17861 tablet_bootstrap.cc:654] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:49.537985 17861 log.cc:824] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:49.537984 17863 log.cc:824] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:49.538682 17861 tablet_bootstrap.cc:492] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece: No bootstrap required, opened a new log
I20260926 13:12:49.538743 17861 ts_tablet_manager.cc:1397] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:12:49.538781 17862 tablet_bootstrap.cc:654] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:49.538955 17863 tablet_bootstrap.cc:492] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db: No bootstrap required, opened a new log
I20260926 13:12:49.539023 17863 ts_tablet_manager.cc:1397] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:12:49.539438 17862 log.cc:824] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:49.540163 17862 tablet_bootstrap.cc:492] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e: No bootstrap required, opened a new log
I20260926 13:12:49.540230 17862 ts_tablet_manager.cc:1397] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 13:12:49.540573 17861 raft_consensus.cc:348] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "b5354496a047489ea8d0b870b5c012db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44973 } } peers { permanent_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36161 } }
I20260926 13:12:49.540717 17861 raft_consensus.cc:374] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:49.540755 17861 raft_consensus.cc:729] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1a82eb5ce543a4bbf2851ace6d5ece, State: Initialized, Role: FOLLOWER
I20260926 13:12:49.540891 17861 consensus_queue.cc:260] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "b5354496a047489ea8d0b870b5c012db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44973 } } peers { permanent_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36161 } }
I20260926 13:12:49.540863 17863 raft_consensus.cc:348] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "b5354496a047489ea8d0b870b5c012db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44973 } } peers { permanent_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36161 } }
I20260926 13:12:49.541026 17863 raft_consensus.cc:374] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:49.541083 17863 raft_consensus.cc:729] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5354496a047489ea8d0b870b5c012db, State: Initialized, Role: FOLLOWER
I20260926 13:12:49.541168 17861 ts_tablet_manager.cc:1428] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 13:12:49.541222 17863 consensus_queue.cc:260] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "b5354496a047489ea8d0b870b5c012db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44973 } } peers { permanent_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36161 } }
I20260926 13:12:49.541296 17576 heartbeater.cc:503] Master 127.0.98.62:41439 was elected leader, sending a full tablet report...
I20260926 13:12:49.541489 17863 ts_tablet_manager.cc:1428] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 13:12:49.541520 17844 heartbeater.cc:503] Master 127.0.98.62:41439 was elected leader, sending a full tablet report...
I20260926 13:12:49.542142 17862 raft_consensus.cc:348] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "b5354496a047489ea8d0b870b5c012db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44973 } } peers { permanent_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36161 } }
I20260926 13:12:49.542304 17862 raft_consensus.cc:374] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:49.542339 17862 raft_consensus.cc:729] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32b378aa6b114b2ea0e03eadc3cd5e3e, State: Initialized, Role: FOLLOWER
I20260926 13:12:49.542435 17862 consensus_queue.cc:260] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "b5354496a047489ea8d0b870b5c012db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44973 } } peers { permanent_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36161 } }
I20260926 13:12:49.542683 17862 ts_tablet_manager.cc:1428] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 13:12:49.542712 17710 heartbeater.cc:503] Master 127.0.98.62:41439 was elected leader, sending a full tablet report...
W20260926 13:12:49.580863 17711 tablet.cc:2369] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:12:49.672525 17577 tablet.cc:2369] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:12:49.706496 17867 raft_consensus.cc:482] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:49.706620 17867 raft_consensus.cc:504] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "b5354496a047489ea8d0b870b5c012db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44973 } } peers { permanent_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36161 } }
I20260926 13:12:49.707033 17867 leader_election.cc:288] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32b378aa6b114b2ea0e03eadc3cd5e3e (127.0.98.2:46037), b5354496a047489ea8d0b870b5c012db (127.0.98.3:44973)
I20260926 13:12:49.708801 17869 raft_consensus.cc:482] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:49.708907 17869 raft_consensus.cc:504] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "b5354496a047489ea8d0b870b5c012db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44973 } } peers { permanent_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36161 } }
I20260926 13:12:49.709259 17869 leader_election.cc:288] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5354496a047489ea8d0b870b5c012db (127.0.98.3:44973), bd1a82eb5ce543a4bbf2851ace6d5ece (127.0.98.1:36161)
I20260926 13:12:49.710933 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.707578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33085 (local address 127.0.98.3:44973)
0926 13:12:49.707628 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.707631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.707639 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:12:49.707667 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.707668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.707676 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.707718 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:49.707835 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.708581 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.709956 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.710547 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.710591 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.710600 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.710803 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.710865 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:12:49.711103 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.707309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54681 (local address 127.0.98.2:46037)
0926 13:12:49.707366 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.707369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.707458 (+ 89us) server_negotiation.cc:407] Connection header received
0926 13:12:49.707584 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.707586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.707598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.707648 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:49.707822 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.708568 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.710065 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.710610 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.710652 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.710661 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.710887 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.711014 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":32}
I20260926 13:12:49.711187 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.707406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44973 (local address 127.0.98.1:33085)
0926 13:12:49.707563 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:49.707573 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:49.707626 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:49.707732 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:49.707732 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:49.707733 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:49.707779 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.707781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.708677 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.708678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.709870 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.709872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.710655 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.710656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.710725 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.710730 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:12:49.711055 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 13:12:49.711104 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":36,"threads_started":1}
I20260926 13:12:49.711320 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.707229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46037 (local address 127.0.98.1:54681)
0926 13:12:49.707362 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:49.707374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:49.707459 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:49.707672 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:49.707673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:49.707675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:49.707752 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.707754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.708656 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.708657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.709953 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.709956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.710715 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.710716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.710788 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.710792 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:12:49.710916 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 13:12:49.711182 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":27,"thread_start_us":81,"threads_started":1}
I20260926 13:12:49.711683 17664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97278d00f6bf43b1abea2175862367b3" candidate_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" is_pre_election: true
I20260926 13:12:49.711664 17798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97278d00f6bf43b1abea2175862367b3" candidate_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5354496a047489ea8d0b870b5c012db" is_pre_election: true
I20260926 13:12:49.711778 17664 raft_consensus.cc:2454] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1a82eb5ce543a4bbf2851ace6d5ece in term 0.
I20260926 13:12:49.711807 17798 raft_consensus.cc:2454] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1a82eb5ce543a4bbf2851ace6d5ece in term 0.
I20260926 13:12:49.711997 17465 leader_election.cc:302] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32b378aa6b114b2ea0e03eadc3cd5e3e, bd1a82eb5ce543a4bbf2851ace6d5ece; no voters:
I20260926 13:12:49.712111 17867 raft_consensus.cc:2781] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:12:49.712158 17867 raft_consensus.cc:482] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:12:49.712178 17867 raft_consensus.cc:3037] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:49.712853 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.709373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44973 (local address 127.0.98.2:36803)
0926 13:12:49.709538 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:49.709547 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:49.709584 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:49.709814 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:49.709815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:49.709817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:49.709871 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.709872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.710815 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.710816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.711993 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.711996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.712603 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.712604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.712670 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.712673 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:49.712715 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 13:12:49.712796 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260926 13:12:49.712865 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.709487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36803 (local address 127.0.98.3:44973)
0926 13:12:49.709650 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.709652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.709660 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:12:49.709681 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.709681 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.709690 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.709924 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:49.709956 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.710731 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.712081 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.712526 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.712585 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.712595 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.712750 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.712788 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260926 13:12:49.713037 17867 raft_consensus.cc:504] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "b5354496a047489ea8d0b870b5c012db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44973 } } peers { permanent_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36161 } }
I20260926 13:12:49.713042 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.709692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38589 (local address 127.0.98.1:36161)
0926 13:12:49.709730 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.709732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.709758 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:12:49.709836 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.709837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.709846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.709870 (+ 24us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:49.710071 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.710701 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.712197 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.712662 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.712737 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.712746 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.712952 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.712990 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:12:49.713176 17867 leader_election.cc:288] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [CANDIDATE]: Term 1 election: Requested vote from peers 32b378aa6b114b2ea0e03eadc3cd5e3e (127.0.98.2:46037), b5354496a047489ea8d0b870b5c012db (127.0.98.3:44973)
I20260926 13:12:49.713261 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.709428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36161 (local address 127.0.98.2:38589)
0926 13:12:49.709701 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:49.709709 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:49.709777 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:49.709932 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:49.709933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:49.709935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:49.710000 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.710001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.710787 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.710788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.712144 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:49.712146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.712781 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.712782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:49.712859 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.712863 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:12:49.713054 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 13:12:49.713102 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":32,"thread_start_us":73,"threads_started":1}
I20260926 13:12:49.713301 17664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97278d00f6bf43b1abea2175862367b3" candidate_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e"
I20260926 13:12:49.713327 17798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97278d00f6bf43b1abea2175862367b3" candidate_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5354496a047489ea8d0b870b5c012db"
I20260926 13:12:49.713379 17664 raft_consensus.cc:3037] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:49.713398 17798 raft_consensus.cc:3037] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:49.713469 17797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97278d00f6bf43b1abea2175862367b3" candidate_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5354496a047489ea8d0b870b5c012db" is_pre_election: true
I20260926 13:12:49.714066 17664 raft_consensus.cc:2454] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1a82eb5ce543a4bbf2851ace6d5ece in term 1.
I20260926 13:12:49.714159 17798 raft_consensus.cc:2454] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1a82eb5ce543a4bbf2851ace6d5ece in term 1.
I20260926 13:12:49.714260 17465 leader_election.cc:302] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32b378aa6b114b2ea0e03eadc3cd5e3e, bd1a82eb5ce543a4bbf2851ace6d5ece; no voters:
I20260926 13:12:49.714282 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97278d00f6bf43b1abea2175862367b3" candidate_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" is_pre_election: true
I20260926 13:12:49.714341 17867 raft_consensus.cc:2781] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:49.714495 17867 raft_consensus.cc:686] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 1 LEADER]: Becoming Leader. State: Replica: bd1a82eb5ce543a4bbf2851ace6d5ece, State: Running, Role: LEADER
I20260926 13:12:49.714581 17867 consensus_queue.cc:237] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } } peers { permanent_uuid: "b5354496a047489ea8d0b870b5c012db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44973 } } peers { permanent_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36161 } }
I20260926 13:12:49.715243 17598 leader_election.cc:302] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32b378aa6b114b2ea0e03eadc3cd5e3e; no voters: b5354496a047489ea8d0b870b5c012db, bd1a82eb5ce543a4bbf2851ace6d5ece
I20260926 13:12:49.715364 17869 raft_consensus.cc:2726] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 13:12:49.715701 17385 catalog_manager.cc:5495] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece reported cstate change: term changed from 0 to 1, leader changed from <none> to bd1a82eb5ce543a4bbf2851ace6d5ece (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5354496a047489ea8d0b870b5c012db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd1a82eb5ce543a4bbf2851ace6d5ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36161 } health_report { overall_health: HEALTHY } } }
W20260926 13:12:49.749823 17845 tablet.cc:2369] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:12:49.757942 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.755650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46572 (local address 127.0.98.1:36161)
0926 13:12:49.755703 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.755705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.755713 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:12:49.755755 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.755756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.755780 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.755839 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:12:49.755990 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.756465 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.757204 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.757391 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.757446 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.757627 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:12:49.757733 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:12:49.757802 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.757840 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.757863 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:12:49.759778 17798 raft_consensus.cc:1261] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db [term 1 FOLLOWER]: Refusing update from remote peer bd1a82eb5ce543a4bbf2851ace6d5ece: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:49.759817 17664 raft_consensus.cc:1261] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [term 1 FOLLOWER]: Refusing update from remote peer bd1a82eb5ce543a4bbf2851ace6d5ece: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:49.760092 17867 consensus_queue.cc:1035] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5354496a047489ea8d0b870b5c012db" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:49.760242 17874 consensus_queue.cc:1035] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "32b378aa6b114b2ea0e03eadc3cd5e3e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:49.762753 17879 mvcc.cc:205] Tried to move back new op lower bound from 7333594602533466112 to 7333594602351767552. Current Snapshot: MvccSnapshot[applied={T|T < 7333594602533466112}]
I20260926 13:12:49.765591 17385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:49.765686 17385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:49.766440 17385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19f2668ace1f4fe5ae29b199c64d2587: Sending DeleteTablet for 3 replicas of tablet 97278d00f6bf43b1abea2175862367b3
I20260926 13:12:49.766810 17778 tablet_service.cc:1514] Processing DeleteTablet for tablet 97278d00f6bf43b1abea2175862367b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41044
I20260926 13:12:49.766887 17510 tablet_service.cc:1514] Processing DeleteTablet for tablet 97278d00f6bf43b1abea2175862367b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46564
I20260926 13:12:49.767092 17889 tablet_replica.cc:330] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece: stopping tablet replica
I20260926 13:12:49.767194 17889 raft_consensus.cc:2229] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:12:49.767287 17889 raft_consensus.cc:2258] T 97278d00f6bf43b1abea2175862367b3 P bd1a82eb5ce543a4bbf2851ace6d5ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:49.767323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 17447
I20260926 13:12:49.767346 17644 tablet_service.cc:1514] Processing DeleteTablet for tablet 97278d00f6bf43b1abea2175862367b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35348
I20260926 13:12:49.767992 17890 tablet_replica.cc:330] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e: stopping tablet replica
I20260926 13:12:49.768133 17890 raft_consensus.cc:2229] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:49.768267 17890 raft_consensus.cc:2258] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:49.768765 17890 ts_tablet_manager.cc:1905] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:49.768848 17888 tablet_replica.cc:330] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db: stopping tablet replica
I20260926 13:12:49.768958 17888 raft_consensus.cc:2229] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:49.769064 17888 raft_consensus.cc:2258] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:49.769559 17888 ts_tablet_manager.cc:1905] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:49.770576 17890 ts_tablet_manager.cc:1918] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:49.770637 17890 log.cc:1196] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/97278d00f6bf43b1abea2175862367b3
I20260926 13:12:49.770864 17890 ts_tablet_manager.cc:1939] T 97278d00f6bf43b1abea2175862367b3 P 32b378aa6b114b2ea0e03eadc3cd5e3e: Deleting consensus metadata
I20260926 13:12:49.771253 17371 catalog_manager.cc:4841] TS 32b378aa6b114b2ea0e03eadc3cd5e3e (127.0.98.2:46037): tablet 97278d00f6bf43b1abea2175862367b3 (table test-table [id=f03f00fd21974f53934abf1e5ae6a91c]) successfully deleted
I20260926 13:12:49.771761 17888 ts_tablet_manager.cc:1918] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:49.771835 17888 log.cc:1196] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/97278d00f6bf43b1abea2175862367b3
I20260926 13:12:49.772075 17888 ts_tablet_manager.cc:1939] T 97278d00f6bf43b1abea2175862367b3 P b5354496a047489ea8d0b870b5c012db: Deleting consensus metadata
I20260926 13:12:49.772670 17371 catalog_manager.cc:4841] TS b5354496a047489ea8d0b870b5c012db (127.0.98.3:44973): tablet 97278d00f6bf43b1abea2175862367b3 (table test-table [id=f03f00fd21974f53934abf1e5ae6a91c]) successfully deleted
W20260926 13:12:49.773406 17370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36161 (error 108)
I20260926 13:12:49.773895 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.773784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36161 (local address 127.0.0.1:46584)
0926 13:12:49.773822 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:49.773860 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 13:12:49.773962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 17581
W20260926 13:12:49.774037 17370 catalog_manager.cc:4584] TS bd1a82eb5ce543a4bbf2851ace6d5ece (127.0.98.1:36161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97278d00f6bf43b1abea2175862367b3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36161: connect: Connection refused (error 111)
I20260926 13:12:49.778712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 17715
I20260926 13:12:49.785338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 17353
2026-09-26T13:12:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2057 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T13:12:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:12:49Z Disabled control of system clock
I20260926 13:12:49.810720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40057
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:12:49.890065 17895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:49.890275 17895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:49.890295 17895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:49.891420 17895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:12:49.891455 17895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:49.891466 17895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:49.891477 17895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:12:49.891489 17895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:12:49.892736 17895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:49.892926 17895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:49.893149 17895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:49.898066 17903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:49.898010 17901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:49.898048 17895 server_base.cc:1034] running on GCE node
W20260926 13:12:49.898013 17900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:49.898531 17895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:49.898821 17895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:49.899997 17895 hybrid_clock.cc:648] HybridClock initialized: now 1790428369899982 us; error 28 us; skew 500 ppm
I20260926 13:12:49.901235 17895 webserver.cc:460] Webserver started at http://127.0.98.62:33035/ using document root <none> and password file <none>
I20260926 13:12:49.901458 17895 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:49.901508 17895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:49.901633 17895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:49.902702 17895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "6c8670cbefea4bef9f7dc70bfbb380c8"
format_stamp: "Formatted at 2026-09-26 13:12:49 on dist-test-slave-2dw5"
server_key: "88954fde28131c22c744d0928d65375b"
server_key_iv: "dcf60f979fd7db5336341471e02615c1"
server_key_version: "encryptionkey@0"
I20260926 13:12:49.903055 17895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c8670cbefea4bef9f7dc70bfbb380c8"
format_stamp: "Formatted at 2026-09-26 13:12:49 on dist-test-slave-2dw5"
server_key: "88954fde28131c22c744d0928d65375b"
server_key_iv: "dcf60f979fd7db5336341471e02615c1"
server_key_version: "encryptionkey@0"
I20260926 13:12:49.904289 17895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:49.905030 17909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:49.905179 17895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:12:49.905260 17895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "6c8670cbefea4bef9f7dc70bfbb380c8"
format_stamp: "Formatted at 2026-09-26 13:12:49 on dist-test-slave-2dw5"
server_key: "88954fde28131c22c744d0928d65375b"
server_key_iv: "dcf60f979fd7db5336341471e02615c1"
server_key_version: "encryptionkey@0"
I20260926 13:12:49.905325 17895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:49.923981 17895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:49.924312 17895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:49.924455 17895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:49.928472 17895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40057
I20260926 13:12:49.928500 17961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40057 every 8 connection(s)
I20260926 13:12:49.928834 17895 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:12:49.929808 17962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:49.932430 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8: Bootstrap starting.
I20260926 13:12:49.932987 17962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:49.933244 17962 log.cc:824] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:49.933916 17962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8: No bootstrap required, opened a new log
I20260926 13:12:49.935070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 17895
I20260926 13:12:49.935186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:12:49.935384 392 external_mini_cluster.cc:1428] Setting key a2bf65f402393608ed6efab8a74f1d71
I20260926 13:12:49.935894 17962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8670cbefea4bef9f7dc70bfbb380c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } }
I20260926 13:12:49.936084 17962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:49.936128 17962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8670cbefea4bef9f7dc70bfbb380c8, State: Initialized, Role: FOLLOWER
I20260926 13:12:49.936257 17962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8670cbefea4bef9f7dc70bfbb380c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } }
I20260926 13:12:49.936331 17962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:12:49.936371 17962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:12:49.936419 17962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:49.937079 17962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8670cbefea4bef9f7dc70bfbb380c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } }
I20260926 13:12:49.937179 17962 leader_election.cc:302] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c8670cbefea4bef9f7dc70bfbb380c8; no voters:
I20260926 13:12:49.937340 17962 leader_election.cc:288] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:12:49.937391 17967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:49.937536 17967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [term 1 LEADER]: Becoming Leader. State: Replica: 6c8670cbefea4bef9f7dc70bfbb380c8, State: Running, Role: LEADER
I20260926 13:12:49.937584 17962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:12:49.937630 17967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8670cbefea4bef9f7dc70bfbb380c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } }
I20260926 13:12:49.938059 17967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c8670cbefea4bef9f7dc70bfbb380c8. Latest consensus state: current_term: 1 leader_uuid: "6c8670cbefea4bef9f7dc70bfbb380c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8670cbefea4bef9f7dc70bfbb380c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } } }
I20260926 13:12:49.938127 17967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:49.938124 17969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c8670cbefea4bef9f7dc70bfbb380c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8670cbefea4bef9f7dc70bfbb380c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40057 } } }
I20260926 13:12:49.938309 17969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8 [sys.catalog]: This master's current role is: LEADER
W20260926 13:12:49.939236 17982 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:12:49.939330 17982 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:12:49.939394 17976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:12:49.939530 17976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:12:49.941205 17976 catalog_manager.cc:1347] Generated new cluster ID: bef75e62c0dd4cc49e9720edb92c7a77
I20260926 13:12:49.941253 17976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:12:49.941560 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:49.935932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.0.98.62:40057)
0926 13:12:49.936147 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:49.936157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:49.936190 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:12:49.936374 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:49.936381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:49.936464 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:49.936590 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:49.936843 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.938458 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.939574 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:49.939998 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:49.940107 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:49.940748 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:49.940752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:49.940753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:49.940964 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:49.941006 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:49.941022 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:49.941081 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:49.941280 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 13:12:49.941359 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260926 13:12:49.945724 17976 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:12:49.946233 17976 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:12:49.957038 17976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8: Generated new TSK 0
I20260926 13:12:49.957261 17976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:12:49.962762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40057
--builtin_ntp_servers=127.0.98.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:12:50.044027 17986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:50.044250 17986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:50.044278 17986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:50.045497 17986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:50.045544 17986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:50.045571 17986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:12:50.047044 17986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:50.047286 17986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:50.047583 17986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:50.053428 17994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:50.053442 17992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:50.053428 17991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:50.053819 17986 server_base.cc:1034] running on GCE node
I20260926 13:12:50.054021 17986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:50.054246 17986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:50.055380 17986 hybrid_clock.cc:648] HybridClock initialized: now 1790428370055364 us; error 29 us; skew 500 ppm
I20260926 13:12:50.056615 17986 webserver.cc:460] Webserver started at http://127.0.98.1:43551/ using document root <none> and password file <none>
I20260926 13:12:50.056825 17986 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:50.056888 17986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:50.057025 17986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:50.058076 17986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "5ae302108e034cb4b27d1ee54ceee5b7"
format_stamp: "Formatted at 2026-09-26 13:12:50 on dist-test-slave-2dw5"
server_key: "603b17acf53dc1210a3331581e588289"
server_key_iv: "106178b6155ead67042c9e2afb59d84a"
server_key_version: "encryptionkey@0"
I20260926 13:12:50.058400 17986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5ae302108e034cb4b27d1ee54ceee5b7"
format_stamp: "Formatted at 2026-09-26 13:12:50 on dist-test-slave-2dw5"
server_key: "603b17acf53dc1210a3331581e588289"
server_key_iv: "106178b6155ead67042c9e2afb59d84a"
server_key_version: "encryptionkey@0"
I20260926 13:12:50.059669 17986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:12:50.060392 18000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:50.060540 17986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:12:50.060619 17986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "5ae302108e034cb4b27d1ee54ceee5b7"
format_stamp: "Formatted at 2026-09-26 13:12:50 on dist-test-slave-2dw5"
server_key: "603b17acf53dc1210a3331581e588289"
server_key_iv: "106178b6155ead67042c9e2afb59d84a"
server_key_version: "encryptionkey@0"
I20260926 13:12:50.060679 17986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:50.067746 17986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:50.068024 17986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:50.068141 17986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:50.068342 17986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:50.068639 17986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:50.068673 17986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:50.068701 17986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:50.068725 17986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:50.074122 17986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38351
I20260926 13:12:50.074183 18113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38351 every 8 connection(s)
I20260926 13:12:50.074491 17986 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:12:50.077412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 17986
I20260926 13:12:50.077495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:12:50.077684 392 external_mini_cluster.cc:1428] Setting key 4a113d86df17eb0b20191b723472a8a3
I20260926 13:12:50.078575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40057
--builtin_ntp_servers=127.0.98.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:12:50.080035 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.075244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48163 (local address 127.0.98.62:40057)
0926 13:12:50.075323 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:50.075325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:50.075532 (+ 207us) server_negotiation.cc:407] Connection header received
0926 13:12:50.076043 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:50.076045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:50.076060 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:50.076110 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:50.076845 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.077358 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.078507 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.078716 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.078778 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.079742 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:50.079743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:50.079743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:50.079748 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:50.079764 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:50.079769 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:50.079825 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:50.079916 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 13:12:50.079950 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:12:50.080199 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.075238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40057 (local address 127.0.98.1:48163)
0926 13:12:50.075418 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:50.075452 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:50.075946 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:50.076179 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:50.076183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:50.076192 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:50.076748 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.076753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.077445 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.077446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.078432 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.078435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.078792 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.078794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.078994 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.079584 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:12:50.079585 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:50.079623 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:12:50.079652 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:50.079834 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:50.079840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:50.079844 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:12:50.079908 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 13:12:50.080006 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260926 13:12:50.080642 18114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40057
I20260926 13:12:50.080747 18114 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:50.080946 18114 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
I20260926 13:12:50.081408 17926 ts_manager.cc:194] Registered new tserver with Master: 5ae302108e034cb4b27d1ee54ceee5b7 (127.0.98.1:38351)
I20260926 13:12:50.081998 17926 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48163
W20260926 13:12:50.158598 18117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:50.158815 18117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:50.158857 18117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:50.160032 18117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:50.160079 18117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:50.160102 18117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:12:50.161443 18117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:50.161700 18117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:50.162014 18117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:50.167032 18122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:50.167193 18117 server_base.cc:1034] running on GCE node
W20260926 13:12:50.167034 18123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:50.167034 18125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:50.167488 18117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:50.167713 18117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:50.168856 18117 hybrid_clock.cc:648] HybridClock initialized: now 1790428370168846 us; error 29 us; skew 500 ppm
I20260926 13:12:50.170083 18117 webserver.cc:460] Webserver started at http://127.0.98.2:46407/ using document root <none> and password file <none>
I20260926 13:12:50.170325 18117 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:50.170374 18117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:50.170499 18117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:50.171622 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "9f0057bc230c4d8d82caeb48aa5d75f9"
format_stamp: "Formatted at 2026-09-26 13:12:50 on dist-test-slave-2dw5"
server_key: "a69ae181715e5a45f84b959a9dca5ba5"
server_key_iv: "baf9fa9693e31f9b8113b6084c987bdc"
server_key_version: "encryptionkey@0"
I20260926 13:12:50.171943 18117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9f0057bc230c4d8d82caeb48aa5d75f9"
format_stamp: "Formatted at 2026-09-26 13:12:50 on dist-test-slave-2dw5"
server_key: "a69ae181715e5a45f84b959a9dca5ba5"
server_key_iv: "baf9fa9693e31f9b8113b6084c987bdc"
server_key_version: "encryptionkey@0"
I20260926 13:12:50.173156 18117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:12:50.173892 18131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:50.174026 18117 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:12:50.174113 18117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "9f0057bc230c4d8d82caeb48aa5d75f9"
format_stamp: "Formatted at 2026-09-26 13:12:50 on dist-test-slave-2dw5"
server_key: "a69ae181715e5a45f84b959a9dca5ba5"
server_key_iv: "baf9fa9693e31f9b8113b6084c987bdc"
server_key_version: "encryptionkey@0"
I20260926 13:12:50.174180 18117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:50.204813 18117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:50.205139 18117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:50.205281 18117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:50.205508 18117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:50.205859 18117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:50.205899 18117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:50.205935 18117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:50.205962 18117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:50.212308 18117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40403
I20260926 13:12:50.212369 18244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40403 every 8 connection(s)
I20260926 13:12:50.212708 18117 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:12:50.213822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 18117
I20260926 13:12:50.213917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:12:50.214092 392 external_mini_cluster.cc:1428] Setting key 8cb0cbab5b74706fd261bfb0b7e0718f
I20260926 13:12:50.214977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40057
--builtin_ntp_servers=127.0.98.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:12:50.218087 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.213434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53567 (local address 127.0.98.62:40057)
0926 13:12:50.213500 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:50.213501 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:50.213728 (+ 227us) server_negotiation.cc:407] Connection header received
0926 13:12:50.214324 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:50.214326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:50.214340 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:50.214403 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:50.215169 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.215676 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.216668 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.216883 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.216949 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.217795 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:50.217797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:50.217797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:50.217802 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:50.217819 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:50.217824 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:50.217878 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:50.217958 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 13:12:50.217991 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:12:50.218284 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.213402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40057 (local address 127.0.98.2:53567)
0926 13:12:50.213592 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:50.213634 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:50.214212 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:50.214475 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:50.214479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:50.214490 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:50.215052 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.215058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.215774 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.215775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.216572 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.216575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.216965 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.216966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.217112 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.217654 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:12:50.217654 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:50.217684 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:12:50.217708 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:50.217883 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:50.217889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:50.217892 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:50.217961 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:12:50.218067 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260926 13:12:50.218787 18245 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40057
I20260926 13:12:50.218905 18245 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:50.219090 18245 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
I20260926 13:12:50.219482 17926 ts_manager.cc:194] Registered new tserver with Master: 9f0057bc230c4d8d82caeb48aa5d75f9 (127.0.98.2:40403)
I20260926 13:12:50.219863 17926 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53567
W20260926 13:12:50.293915 18248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:50.294126 18248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:50.294147 18248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:50.295339 18248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:50.295378 18248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:50.295403 18248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:12:50.296764 18248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:50.296996 18248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:50.297282 18248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:50.302351 18253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:50.302346 18254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:50.302390 18248 server_base.cc:1034] running on GCE node
W20260926 13:12:50.302352 18256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:50.302754 18248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:50.303035 18248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:50.304189 18248 hybrid_clock.cc:648] HybridClock initialized: now 1790428370304170 us; error 38 us; skew 500 ppm
I20260926 13:12:50.305496 18248 webserver.cc:460] Webserver started at http://127.0.98.3:46009/ using document root <none> and password file <none>
I20260926 13:12:50.305732 18248 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:50.305774 18248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:50.305871 18248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:50.306928 18248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "0b5b61e1aacc4b7db2366a3315fdb16d"
format_stamp: "Formatted at 2026-09-26 13:12:50 on dist-test-slave-2dw5"
server_key: "74bc7f6135b8e31646ecb6e87b147277"
server_key_iv: "afea03bffc47334fa22a474e71f9d0d2"
server_key_version: "encryptionkey@0"
I20260926 13:12:50.307224 18248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0b5b61e1aacc4b7db2366a3315fdb16d"
format_stamp: "Formatted at 2026-09-26 13:12:50 on dist-test-slave-2dw5"
server_key: "74bc7f6135b8e31646ecb6e87b147277"
server_key_iv: "afea03bffc47334fa22a474e71f9d0d2"
server_key_version: "encryptionkey@0"
I20260926 13:12:50.308429 18248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:50.309206 18262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:50.309336 18248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:50.309413 18248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "0b5b61e1aacc4b7db2366a3315fdb16d"
format_stamp: "Formatted at 2026-09-26 13:12:50 on dist-test-slave-2dw5"
server_key: "74bc7f6135b8e31646ecb6e87b147277"
server_key_iv: "afea03bffc47334fa22a474e71f9d0d2"
server_key_version: "encryptionkey@0"
I20260926 13:12:50.309481 18248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:50.325133 18248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:50.325439 18248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:50.325572 18248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:50.325773 18248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:50.326097 18248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:50.326131 18248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:50.326160 18248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:50.326177 18248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:50.331625 18248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46375
I20260926 13:12:50.331681 18375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46375 every 8 connection(s)
I20260926 13:12:50.331977 18248 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:12:50.337152 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.332713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49241 (local address 127.0.98.62:40057)
0926 13:12:50.332774 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:50.332776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:50.332969 (+ 193us) server_negotiation.cc:407] Connection header received
0926 13:12:50.333549 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:50.333550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:50.333560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:50.333637 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:50.334248 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.334748 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.335878 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.336096 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.336159 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.336913 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:50.336915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:50.336915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:50.336920 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:50.336935 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:50.336939 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:50.336975 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:50.337033 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 13:12:50.337059 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:12:50.337285 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.332689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40057 (local address 127.0.98.3:49241)
0926 13:12:50.332859 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:50.332899 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:50.333468 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:50.333692 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:50.333695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:50.333704 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:50.334186 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.334189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.334813 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.334813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.335736 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.335739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.336215 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.336216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.336334 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.336793 (+ 459us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:12:50.336793 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:50.336819 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:12:50.336842 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:50.336982 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:50.336984 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:50.336986 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:12:50.337029 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 13:12:50.337100 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260926 13:12:50.337728 18376 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40057
I20260926 13:12:50.337809 18376 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:50.337980 18376 heartbeater.cc:511] Master 127.0.98.62:40057 requested a full tablet report, sending...
I20260926 13:12:50.338294 17926 ts_manager.cc:194] Registered new tserver with Master: 0b5b61e1aacc4b7db2366a3315fdb16d (127.0.98.3:46375)
I20260926 13:12:50.338759 17926 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49241
I20260926 13:12:50.339406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 18248
I20260926 13:12:50.339483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:12:50.339653 392 external_mini_cluster.cc:1428] Setting key 5e96554b1f92c93c6cc69cc2513e585d
I20260926 13:12:50.340385 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:12:50.344395 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.341674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.0.98.62:40057)
0926 13:12:50.341723 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:50.341725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:50.341876 (+ 151us) server_negotiation.cc:407] Connection header received
0926 13:12:50.341935 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:50.341936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:50.341948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:50.341992 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:50.342130 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.342614 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.343375 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.343568 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.343620 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.344177 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:50.344178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:50.344178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:50.344182 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:50.344196 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:50.344200 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:50.344242 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:50.344298 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 13:12:50.344322 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 13:12:50.345659 17926 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:12:50.347191 17926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:12:50.352819 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.349454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51586 (local address 127.0.98.1:38351)
0926 13:12:50.349735 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:50.349738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:50.349746 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:12:50.349784 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:50.349786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:50.349799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:50.349872 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:50.349941 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.350990 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.351869 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.352457 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.352525 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.352611 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:50.352705 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 13:12:50.352740 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":78,"threads_started":1}
I20260926 13:12:50.352825 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.349240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38351 (local address 127.0.0.1:51586)
0926 13:12:50.349561 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:50.349568 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:50.349604 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:50.349870 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:50.349870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:50.349872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:50.349895 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.349896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.351054 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.351055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.351792 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.351794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.352524 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.352525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.352613 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.352626 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 13:12:50.352691 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:12:50.352738 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260926 13:12:50.353346 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.349179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.0.98.3:46375)
0926 13:12:50.349310 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:50.349314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:50.349477 (+ 163us) server_negotiation.cc:407] Connection header received
0926 13:12:50.349541 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:50.349544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:50.349558 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:50.349630 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:50.349783 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.350917 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.352219 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.352909 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.352973 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.353050 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:50.353230 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 13:12:50.353268 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260926 13:12:50.353363 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.349015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40403 (local address 127.0.0.1:40008)
0926 13:12:50.349353 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:50.349383 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:50.349456 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:50.349723 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:50.349724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:50.349725 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:50.349754 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.349755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.350822 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.350823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.352136 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.352138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.353140 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.353140 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.353187 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.353190 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:50.353261 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:12:50.353299 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":31,"thread_start_us":72,"threads_started":1}
I20260926 13:12:50.353633 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.349016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46375 (local address 127.0.0.1:34872)
0926 13:12:50.349346 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:50.349383 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:50.349456 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:50.349643 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:50.349646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:50.349648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:50.349710 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.349712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.350991 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.350992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.352135 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.352138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.353082 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.353082 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.353142 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.353146 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:12:50.353525 (+ 379us) client_negotiation.cc:240] Negotiation successful
0926 13:12:50.353570 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260926 13:12:50.353828 18048 tablet_service.cc:1467] Processing CreateTablet for tablet 4c605855c3b24c8389ea40aefe690be2 (DEFAULT_TABLE table=test-table [id=89afcbbe3a734b69a74dc5276e1909c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:50.354151 18048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c605855c3b24c8389ea40aefe690be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:50.354207 18310 tablet_service.cc:1467] Processing CreateTablet for tablet 4c605855c3b24c8389ea40aefe690be2 (DEFAULT_TABLE table=test-table [id=89afcbbe3a734b69a74dc5276e1909c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:50.354445 18310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c605855c3b24c8389ea40aefe690be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:50.355078 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.349090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.0.98.2:40403)
0926 13:12:50.349250 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:50.349254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:50.349449 (+ 195us) server_negotiation.cc:407] Connection header received
0926 13:12:50.349571 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:50.349574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:50.349590 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:50.349640 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:50.349831 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.350752 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.352292 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.352984 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.353021 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.353085 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:50.353282 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 13:12:50.353324 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260926 13:12:50.355870 18179 tablet_service.cc:1467] Processing CreateTablet for tablet 4c605855c3b24c8389ea40aefe690be2 (DEFAULT_TABLE table=test-table [id=89afcbbe3a734b69a74dc5276e1909c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:50.356184 18179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c605855c3b24c8389ea40aefe690be2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:50.357038 18390 tablet_bootstrap.cc:492] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d: Bootstrap starting.
I20260926 13:12:50.357360 18391 tablet_bootstrap.cc:492] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7: Bootstrap starting.
I20260926 13:12:50.357791 18390 tablet_bootstrap.cc:654] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:50.357944 18391 tablet_bootstrap.cc:654] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:50.358119 18390 log.cc:824] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:50.358217 18391 log.cc:824] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:50.358832 18390 tablet_bootstrap.cc:492] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d: No bootstrap required, opened a new log
I20260926 13:12:50.358865 18391 tablet_bootstrap.cc:492] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7: No bootstrap required, opened a new log
I20260926 13:12:50.358891 18394 tablet_bootstrap.cc:492] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9: Bootstrap starting.
I20260926 13:12:50.358915 18391 ts_tablet_manager.cc:1397] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:12:50.358915 18390 ts_tablet_manager.cc:1397] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:12:50.359390 18394 tablet_bootstrap.cc:654] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:50.359637 18394 log.cc:824] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:50.360183 18394 tablet_bootstrap.cc:492] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9: No bootstrap required, opened a new log
I20260926 13:12:50.360234 18394 ts_tablet_manager.cc:1397] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:50.361275 18390 raft_consensus.cc:348] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46375 } } peers { permanent_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38351 } } peers { permanent_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40403 } }
I20260926 13:12:50.361275 18391 raft_consensus.cc:348] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46375 } } peers { permanent_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38351 } } peers { permanent_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40403 } }
I20260926 13:12:50.361442 18391 raft_consensus.cc:374] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:50.361442 18390 raft_consensus.cc:374] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:50.361476 18390 raft_consensus.cc:729] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5b61e1aacc4b7db2366a3315fdb16d, State: Initialized, Role: FOLLOWER
I20260926 13:12:50.361476 18391 raft_consensus.cc:729] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ae302108e034cb4b27d1ee54ceee5b7, State: Initialized, Role: FOLLOWER
I20260926 13:12:50.361617 18391 consensus_queue.cc:260] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46375 } } peers { permanent_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38351 } } peers { permanent_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40403 } }
I20260926 13:12:50.361724 18390 consensus_queue.cc:260] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46375 } } peers { permanent_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38351 } } peers { permanent_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40403 } }
I20260926 13:12:50.361881 18391 ts_tablet_manager.cc:1428] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:12:50.361944 18390 ts_tablet_manager.cc:1428] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:12:50.361954 18114 heartbeater.cc:503] Master 127.0.98.62:40057 was elected leader, sending a full tablet report...
I20260926 13:12:50.362073 18376 heartbeater.cc:503] Master 127.0.98.62:40057 was elected leader, sending a full tablet report...
I20260926 13:12:50.362746 18394 raft_consensus.cc:348] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46375 } } peers { permanent_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38351 } } peers { permanent_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40403 } }
I20260926 13:12:50.362933 18394 raft_consensus.cc:374] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:50.362975 18394 raft_consensus.cc:729] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0057bc230c4d8d82caeb48aa5d75f9, State: Initialized, Role: FOLLOWER
I20260926 13:12:50.363082 18394 consensus_queue.cc:260] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46375 } } peers { permanent_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38351 } } peers { permanent_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40403 } }
I20260926 13:12:50.363312 18394 ts_tablet_manager.cc:1428] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:12:50.363488 18245 heartbeater.cc:503] Master 127.0.98.62:40057 was elected leader, sending a full tablet report...
W20260926 13:12:50.463290 18246 tablet.cc:2369] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:12:50.535171 18396 raft_consensus.cc:482] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:50.535342 18396 raft_consensus.cc:504] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46375 } } peers { permanent_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38351 } } peers { permanent_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40403 } }
I20260926 13:12:50.535820 18396 leader_election.cc:288] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b5b61e1aacc4b7db2366a3315fdb16d (127.0.98.3:46375), 9f0057bc230c4d8d82caeb48aa5d75f9 (127.0.98.2:40403)
I20260926 13:12:50.540158 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.536204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60531 (local address 127.0.98.2:40403)
0926 13:12:50.536255 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:50.536258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:50.536270 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:12:50.536306 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:50.536307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:50.536316 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:50.536386 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:50.536586 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.537330 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.539125 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.539727 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.539806 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.539817 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:50.540022 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 13:12:50.540073 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:12:50.540158 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.536201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39255 (local address 127.0.98.3:46375)
0926 13:12:50.536256 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:50.536258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:50.536446 (+ 188us) server_negotiation.cc:407] Connection header received
0926 13:12:50.536481 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:50.536483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:50.536490 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:50.536533 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:50.536695 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.537401 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.539124 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.539716 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.539806 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.539817 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:50.540057 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 13:12:50.540097 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:12:50.540180 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.536074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40403 (local address 127.0.98.1:60531)
0926 13:12:50.536142 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:50.536158 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:50.536215 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:50.536412 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:50.536413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:50.536415 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:50.536505 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.536508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.537457 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.537459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.539000 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.539004 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.539834 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.539835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.539924 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.539931 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:12:50.539996 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:12:50.540057 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":44}
I20260926 13:12:50.540181 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.536124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46375 (local address 127.0.98.1:39255)
0926 13:12:50.536332 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:50.536340 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:50.536396 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:50.536551 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:50.536552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:50.536553 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:50.536629 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.536630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.537495 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.537496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.539022 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:50.539024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.539834 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.539835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:50.539927 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.539931 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:12:50.539996 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:12:50.540053 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":42,"thread_start_us":119,"threads_started":1}
I20260926 13:12:50.540657 18199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c605855c3b24c8389ea40aefe690be2" candidate_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9" is_pre_election: true
I20260926 13:12:50.540921 18199 raft_consensus.cc:2454] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ae302108e034cb4b27d1ee54ceee5b7 in term 0.
I20260926 13:12:50.540879 18330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c605855c3b24c8389ea40aefe690be2" candidate_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d" is_pre_election: true
I20260926 13:12:50.541066 18330 raft_consensus.cc:2454] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ae302108e034cb4b27d1ee54ceee5b7 in term 0.
I20260926 13:12:50.541257 18004 leader_election.cc:302] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ae302108e034cb4b27d1ee54ceee5b7, 9f0057bc230c4d8d82caeb48aa5d75f9; no voters:
I20260926 13:12:50.541440 18396 raft_consensus.cc:2781] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:12:50.541531 18396 raft_consensus.cc:482] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:12:50.541563 18396 raft_consensus.cc:3037] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:50.542481 18396 raft_consensus.cc:504] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46375 } } peers { permanent_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38351 } } peers { permanent_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40403 } }
I20260926 13:12:50.542637 18396 leader_election.cc:288] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [CANDIDATE]: Term 1 election: Requested vote from peers 0b5b61e1aacc4b7db2366a3315fdb16d (127.0.98.3:46375), 9f0057bc230c4d8d82caeb48aa5d75f9 (127.0.98.2:40403)
I20260926 13:12:50.542876 18330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c605855c3b24c8389ea40aefe690be2" candidate_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d"
I20260926 13:12:50.542909 18199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c605855c3b24c8389ea40aefe690be2" candidate_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9"
I20260926 13:12:50.542984 18330 raft_consensus.cc:3037] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:50.543053 18199 raft_consensus.cc:3037] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:50.543902 18330 raft_consensus.cc:2454] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ae302108e034cb4b27d1ee54ceee5b7 in term 1.
I20260926 13:12:50.543964 18199 raft_consensus.cc:2454] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ae302108e034cb4b27d1ee54ceee5b7 in term 1.
I20260926 13:12:50.544147 18003 leader_election.cc:302] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5b61e1aacc4b7db2366a3315fdb16d, 5ae302108e034cb4b27d1ee54ceee5b7; no voters:
I20260926 13:12:50.544323 18396 raft_consensus.cc:2781] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:50.544531 18396 raft_consensus.cc:686] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 1 LEADER]: Becoming Leader. State: Replica: 5ae302108e034cb4b27d1ee54ceee5b7, State: Running, Role: LEADER
I20260926 13:12:50.544644 18396 consensus_queue.cc:237] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46375 } } peers { permanent_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38351 } } peers { permanent_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40403 } }
I20260926 13:12:50.545691 17925 catalog_manager.cc:5495] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ae302108e034cb4b27d1ee54ceee5b7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ae302108e034cb4b27d1ee54ceee5b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40403 } health_report { overall_health: UNKNOWN } } }
W20260926 13:12:50.575206 18115 tablet.cc:2369] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:12:50.580029 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.577379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.0.98.1:38351)
0926 13:12:50.577453 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:50.577456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:50.577481 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:12:50.577525 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:50.577526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:50.577548 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:50.577631 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:12:50.577849 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.578432 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.579259 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:50.579469 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:50.579528 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:50.579661 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:12:50.579798 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:12:50.579866 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:50.579886 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 13:12:50.579906 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:12:50.582233 18199 raft_consensus.cc:1261] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9 [term 1 FOLLOWER]: Refusing update from remote peer 5ae302108e034cb4b27d1ee54ceee5b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:50.582285 18330 raft_consensus.cc:1261] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d [term 1 FOLLOWER]: Refusing update from remote peer 5ae302108e034cb4b27d1ee54ceee5b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 13:12:50.582527 18377 tablet.cc:2369] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:12:50.582717 18396 consensus_queue.cc:1035] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5b61e1aacc4b7db2366a3315fdb16d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:50.582980 18400 consensus_queue.cc:1035] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f0057bc230c4d8d82caeb48aa5d75f9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:50.585793 18403 mvcc.cc:205] Tried to move back new op lower bound from 7333594605901492224 to 7333594605752713216. Current Snapshot: MvccSnapshot[applied={T|T < 7333594605901492224}]
I20260926 13:12:50.586213 18404 mvcc.cc:205] Tried to move back new op lower bound from 7333594605901492224 to 7333594605752713216. Current Snapshot: MvccSnapshot[applied={T|T < 7333594605901492224}]
I20260926 13:12:50.589203 17925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:50.589321 17925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:50.590430 17925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c8670cbefea4bef9f7dc70bfbb380c8: Sending DeleteTablet for 3 replicas of tablet 4c605855c3b24c8389ea40aefe690be2
I20260926 13:12:50.590819 18310 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c605855c3b24c8389ea40aefe690be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:50 UTC) from {username='slave'} at 127.0.0.1:34872
I20260926 13:12:50.590895 18048 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c605855c3b24c8389ea40aefe690be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:50 UTC) from {username='slave'} at 127.0.0.1:51586
I20260926 13:12:50.590924 18179 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c605855c3b24c8389ea40aefe690be2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:50 UTC) from {username='slave'} at 127.0.0.1:40008
I20260926 13:12:50.591262 18414 tablet_replica.cc:330] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9: stopping tablet replica
I20260926 13:12:50.591395 18414 raft_consensus.cc:2229] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:50.591531 18414 raft_consensus.cc:2258] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:50.591636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 17986
I20260926 13:12:50.591728 18413 tablet_replica.cc:330] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7: stopping tablet replica
I20260926 13:12:50.591815 18413 raft_consensus.cc:2229] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:12:50.591951 18413 raft_consensus.cc:2258] T 4c605855c3b24c8389ea40aefe690be2 P 5ae302108e034cb4b27d1ee54ceee5b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:50.592379 18412 tablet_replica.cc:330] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d: stopping tablet replica
I20260926 13:12:50.592515 18412 raft_consensus.cc:2229] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:50.592631 18412 raft_consensus.cc:2258] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:50.592713 18414 ts_tablet_manager.cc:1905] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:50.593116 18412 ts_tablet_manager.cc:1905] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:50.594704 18414 ts_tablet_manager.cc:1918] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:50.594964 18412 ts_tablet_manager.cc:1918] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:50.594879 18414 log.cc:1196] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/4c605855c3b24c8389ea40aefe690be2
I20260926 13:12:50.595050 18412 log.cc:1196] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/4c605855c3b24c8389ea40aefe690be2
I20260926 13:12:50.595345 18412 ts_tablet_manager.cc:1939] T 4c605855c3b24c8389ea40aefe690be2 P 0b5b61e1aacc4b7db2366a3315fdb16d: Deleting consensus metadata
I20260926 13:12:50.595353 18414 ts_tablet_manager.cc:1939] T 4c605855c3b24c8389ea40aefe690be2 P 9f0057bc230c4d8d82caeb48aa5d75f9: Deleting consensus metadata
I20260926 13:12:50.595932 17913 catalog_manager.cc:4841] TS 9f0057bc230c4d8d82caeb48aa5d75f9 (127.0.98.2:40403): tablet 4c605855c3b24c8389ea40aefe690be2 (table test-table [id=89afcbbe3a734b69a74dc5276e1909c7]) successfully deleted
I20260926 13:12:50.596165 17912 catalog_manager.cc:4841] TS 0b5b61e1aacc4b7db2366a3315fdb16d (127.0.98.3:46375): tablet 4c605855c3b24c8389ea40aefe690be2 (table test-table [id=89afcbbe3a734b69a74dc5276e1909c7]) successfully deleted
W20260926 13:12:50.599380 17912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38351 (error 108)
I20260926 13:12:50.599716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 18117
I20260926 13:12:50.600605 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:50.599856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38351 (local address 127.0.0.1:51600)
0926 13:12:50.600498 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:50.600560 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614}
W20260926 13:12:50.600814 17912 catalog_manager.cc:4584] TS 5ae302108e034cb4b27d1ee54ceee5b7 (127.0.98.1:38351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c605855c3b24c8389ea40aefe690be2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38351: connect: Connection refused (error 111)
I20260926 13:12:50.605892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 18248
I20260926 13:12:50.611168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 17895
2026-09-26T13:12:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (826 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:12:50 dist-test-slave-2dw5 krb5kdc[18418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:12:50 dist-test-slave-2dw5 krb5kdc[18418](info): set up 2 sockets
Sep 26 13:12:50 dist-test-slave-2dw5 krb5kdc[18418](info): commencing operation
krb5kdc: starting...
W20260926 13:12:51.566737 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:12:51 dist-test-slave-2dw5 krb5kdc[18418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:12:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:12:51.685247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45963
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:51.766561 18431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:51.766812 18431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:51.766875 18431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:51.768227 18431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:12:51.768298 18431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:51.768312 18431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:51.768325 18431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:12:51.768335 18431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:12:51.770087 18431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:51.770362 18431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:51.770587 18431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:51.775597 18436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:51.775650 18437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:51.775635 18439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:51.775826 18431 server_base.cc:1034] running on GCE node
I20260926 13:12:51.776108 18431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:51.776376 18431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:51.777580 18431 hybrid_clock.cc:648] HybridClock initialized: now 1790428371777560 us; error 39 us; skew 500 ppm
Sep 26 13:12:51 dist-test-slave-2dw5 krb5kdc[18418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:51.780254 18431 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:12:51.780771 18431 webserver.cc:460] Webserver started at http://127.0.98.62:44995/ using document root <none> and password file <none>
I20260926 13:12:51.781033 18431 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:51.781085 18431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:51.781211 18431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:51.782274 18431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "2b2e2d3f96bd4a7a86ab79c912600f2d"
format_stamp: "Formatted at 2026-09-26 13:12:51 on dist-test-slave-2dw5"
server_key: "72608292afb8e25a0360af9988099aac"
server_key_iv: "ba3dedfd3133c097c60d0126a12a50be"
server_key_version: "encryptionkey@0"
I20260926 13:12:51.782608 18431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b2e2d3f96bd4a7a86ab79c912600f2d"
format_stamp: "Formatted at 2026-09-26 13:12:51 on dist-test-slave-2dw5"
server_key: "72608292afb8e25a0360af9988099aac"
server_key_iv: "ba3dedfd3133c097c60d0126a12a50be"
server_key_version: "encryptionkey@0"
I20260926 13:12:51.784003 18431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:12:51.784806 18446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:51.785010 18431 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:51.785099 18431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "2b2e2d3f96bd4a7a86ab79c912600f2d"
format_stamp: "Formatted at 2026-09-26 13:12:51 on dist-test-slave-2dw5"
server_key: "72608292afb8e25a0360af9988099aac"
server_key_iv: "ba3dedfd3133c097c60d0126a12a50be"
server_key_version: "encryptionkey@0"
I20260926 13:12:51.785173 18431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:51.796620 18431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:51.798725 18431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:51.798924 18431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:51.802704 18431 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45963
I20260926 13:12:51.802744 18498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45963 every 8 connection(s)
I20260926 13:12:51.803117 18431 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:12:51.803965 18499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:51.806191 18499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d: Bootstrap starting.
I20260926 13:12:51.806792 18499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:51.807096 18499 log.cc:824] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:51.807770 18499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d: No bootstrap required, opened a new log
I20260926 13:12:51.809329 18499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2e2d3f96bd4a7a86ab79c912600f2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45963 } }
I20260926 13:12:51.809520 18499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:51.809547 18499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b2e2d3f96bd4a7a86ab79c912600f2d, State: Initialized, Role: FOLLOWER
I20260926 13:12:51.809688 18499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2e2d3f96bd4a7a86ab79c912600f2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45963 } }
I20260926 13:12:51.809749 18499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:12:51.809770 18499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:12:51.809815 18499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:51.809979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 18431
I20260926 13:12:51.810077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:12:51.810299 392 external_mini_cluster.cc:1428] Setting key 584aa8b88592c870294a85b3a223b086
I20260926 13:12:51.810393 18499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2e2d3f96bd4a7a86ab79c912600f2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45963 } }
I20260926 13:12:51.810516 18499 leader_election.cc:302] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b2e2d3f96bd4a7a86ab79c912600f2d; no voters:
I20260926 13:12:51.810679 18499 leader_election.cc:288] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:12:51.810751 18502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:51.810941 18502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [term 1 LEADER]: Becoming Leader. State: Replica: 2b2e2d3f96bd4a7a86ab79c912600f2d, State: Running, Role: LEADER
I20260926 13:12:51.810956 18499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:12:51.811108 18502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2e2d3f96bd4a7a86ab79c912600f2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45963 } }
I20260926 13:12:51.811513 18502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b2e2d3f96bd4a7a86ab79c912600f2d. Latest consensus state: current_term: 1 leader_uuid: "2b2e2d3f96bd4a7a86ab79c912600f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2e2d3f96bd4a7a86ab79c912600f2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45963 } } }
I20260926 13:12:51.811518 18503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b2e2d3f96bd4a7a86ab79c912600f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2e2d3f96bd4a7a86ab79c912600f2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45963 } } }
I20260926 13:12:51.811685 18503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:51.811666 18502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:51.812389 18513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:12:51.814060 18513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:12:51.815693 18513 catalog_manager.cc:1347] Generated new cluster ID: 0b08a660e98442d799c2e3061898e876
I20260926 13:12:51.815745 18513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:12:51 dist-test-slave-2dw5 krb5kdc[18418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:51.821537 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:51.810801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.0.98.62:45963)
0926 13:12:51.811025 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:51.811034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:51.811313 (+ 279us) server_negotiation.cc:407] Connection header received
0926 13:12:51.811431 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:51.811439 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:51.811495 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:51.811620 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:51.812530 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:51.813495 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:51.814480 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:51.815106 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:51.815216 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:51.817995 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:51.817999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:51.817999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:51.818012 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:51.819935 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:51.820356 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:51.820358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:51.820359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:51.820470 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:51.820656 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:51.820657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:51.820657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:51.820951 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:51.821095 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:51.821190 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 13:12:51.821329 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":119,"threads_started":1}
I20260926 13:12:51.831632 18513 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:12:51.832201 18513 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:12:51.836069 18513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d: Generated new TSK 0
I20260926 13:12:51.836241 18513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:12:51.862195 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45963
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:51.944177 18525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:51.944425 18525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:51.944459 18525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:51.945716 18525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:51.945765 18525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:51.945803 18525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:12:51.947453 18525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:51.947744 18525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:51.947962 18525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:51.953270 18533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:51.953262 18530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:51.953313 18525 server_base.cc:1034] running on GCE node
W20260926 13:12:51.953218 18531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:51.953704 18525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:51.953933 18525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:51.955071 18525 hybrid_clock.cc:648] HybridClock initialized: now 1790428371955057 us; error 29 us; skew 500 ppm
Sep 26 13:12:51 dist-test-slave-2dw5 krb5kdc[18418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:51.957507 18525 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:12:51.957954 18525 webserver.cc:460] Webserver started at http://127.0.98.1:34985/ using document root <none> and password file <none>
I20260926 13:12:51.958142 18525 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:51.958182 18525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:51.958268 18525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:51.959336 18525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "cd42ce894a7e417ca6cb1635119bd6ac"
format_stamp: "Formatted at 2026-09-26 13:12:51 on dist-test-slave-2dw5"
server_key: "554708b035ea149ba7fd45e7c28ae9b5"
server_key_iv: "6d4e6df14be6d3475b123e947a5e7363"
server_key_version: "encryptionkey@0"
I20260926 13:12:51.959632 18525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cd42ce894a7e417ca6cb1635119bd6ac"
format_stamp: "Formatted at 2026-09-26 13:12:51 on dist-test-slave-2dw5"
server_key: "554708b035ea149ba7fd45e7c28ae9b5"
server_key_iv: "6d4e6df14be6d3475b123e947a5e7363"
server_key_version: "encryptionkey@0"
I20260926 13:12:51.961104 18525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:12:51.961788 18540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:51.961941 18525 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:12:51.962018 18525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "cd42ce894a7e417ca6cb1635119bd6ac"
format_stamp: "Formatted at 2026-09-26 13:12:51 on dist-test-slave-2dw5"
server_key: "554708b035ea149ba7fd45e7c28ae9b5"
server_key_iv: "6d4e6df14be6d3475b123e947a5e7363"
server_key_version: "encryptionkey@0"
I20260926 13:12:51.962075 18525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:51.972134 18525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:51.974104 18525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:51.974239 18525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:51.974433 18525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:51.974695 18525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:51.974725 18525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:51.974745 18525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:51.974762 18525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:51.980521 18525 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39117
I20260926 13:12:51.980683 18653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39117 every 8 connection(s)
I20260926 13:12:51.980898 18525 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:12:51 dist-test-slave-2dw5 krb5kdc[18418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:51.987447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 18525
I20260926 13:12:51.987542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:12:51.987749 392 external_mini_cluster.cc:1428] Setting key 7f6d229a1fc03eb18dd76fcde8a0c39f
I20260926 13:12:51.990553 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:51.981685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59247 (local address 127.0.98.62:45963)
0926 13:12:51.981768 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:51.981771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:51.981942 (+ 171us) server_negotiation.cc:407] Connection header received
0926 13:12:51.982897 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:51.982899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:51.982914 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:51.982957 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:51.983593 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:51.984090 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:51.985078 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:51.985279 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:51.985335 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:51.987732 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:51.987733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:51.987734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:51.987741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:51.989054 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:51.989441 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:51.989443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:51.989443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:51.989496 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:51.989810 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:51.989811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:51.989811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:51.989945 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:51.990009 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:51.990384 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 13:12:51.990473 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:12:51.990677 18656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:51.981649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45963 (local address 127.0.98.1:59247)
0926 13:12:51.981816 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:51.981855 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:51.982777 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:51.983046 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:51.983050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:51.983308 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:51.983499 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:51.983502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:51.984175 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:51.984176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:51.984988 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:51.984990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:51.985351 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:51.985353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:51.985478 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:51.985980 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:51.985988 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:51.987614 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:51.989139 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:51.989148 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:51.989150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:51.989375 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:51.989577 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:51.989578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:51.989578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:51.989746 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:51.990032 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:51.990038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:51.990245 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 13:12:51.990345 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 13:12:51.990489 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260926 13:12:51.991146 18654 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45963
I20260926 13:12:51.991246 18654 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:51.991417 18654 heartbeater.cc:511] Master 127.0.98.62:45963 requested a full tablet report, sending...
I20260926 13:12:51.991930 18463 ts_manager.cc:194] Registered new tserver with Master: cd42ce894a7e417ca6cb1635119bd6ac (127.0.98.1:39117)
I20260926 13:12:51.992794 18463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59247
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:12:52.014109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45963
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:52.094733 18659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:52.094990 18659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:52.095031 18659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:52.096244 18659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:52.096289 18659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:52.096324 18659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:12:52.097842 18659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:52.098068 18659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:52.098278 18659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:52.103366 18667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:52.103389 18664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:52.103456 18665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:52.103605 18659 server_base.cc:1034] running on GCE node
I20260926 13:12:52.103820 18659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:52.104045 18659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:52.105182 18659 hybrid_clock.cc:648] HybridClock initialized: now 1790428372105170 us; error 29 us; skew 500 ppm
Sep 26 13:12:52 dist-test-slave-2dw5 krb5kdc[18418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:52.107807 18659 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:12:52.108227 18659 webserver.cc:460] Webserver started at http://127.0.98.2:40757/ using document root <none> and password file <none>
I20260926 13:12:52.108472 18659 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:52.108525 18659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:52.108644 18659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:52.109685 18659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "c8138574b1d64c24a2565c0008426351"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "f0bf4d74c63f9b1e57386162b00fca6b"
server_key_iv: "d150ac078e2918519b54d2b531b0f58d"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.110013 18659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c8138574b1d64c24a2565c0008426351"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "f0bf4d74c63f9b1e57386162b00fca6b"
server_key_iv: "d150ac078e2918519b54d2b531b0f58d"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.111397 18659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:12:52.112104 18674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:52.112385 18659 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:12:52.112475 18659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "c8138574b1d64c24a2565c0008426351"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "f0bf4d74c63f9b1e57386162b00fca6b"
server_key_iv: "d150ac078e2918519b54d2b531b0f58d"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.112545 18659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:52.137089 18659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:52.139246 18659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:52.139374 18659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:52.139602 18659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:52.139909 18659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:52.139942 18659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:52.139971 18659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:52.139994 18659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:52.145615 18659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34701
I20260926 13:12:52.145749 18787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34701 every 8 connection(s)
I20260926 13:12:52.145999 18659 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:12:52.149281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 18659
I20260926 13:12:52.149413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:12:52.149621 392 external_mini_cluster.cc:1428] Setting key da95675eec15b1347d124b489a25e041
Sep 26 13:12:52 dist-test-slave-2dw5 krb5kdc[18418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:52.156170 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.146720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53073 (local address 127.0.98.62:45963)
0926 13:12:52.146783 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:52.146785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:52.146957 (+ 172us) server_negotiation.cc:407] Connection header received
0926 13:12:52.147881 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:52.147883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:52.147894 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:52.147955 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:52.148536 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.149049 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.150317 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.150545 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.150615 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.153058 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:52.153059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:52.153059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:52.153068 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:52.154574 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:52.155038 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:52.155039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:52.155039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:52.155103 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:52.155415 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:52.155416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:52.155416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:52.155565 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:52.155632 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:52.155973 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 13:12:52.156073 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:12:52.156414 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.146668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45963 (local address 127.0.98.2:53073)
0926 13:12:52.146823 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:52.146892 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:52.147780 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:52.148008 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:52.148012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:52.148269 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:52.148463 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.148467 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.149154 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.149156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.150238 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.150241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.150639 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.150640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.150757 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.151360 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:52.151369 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:52.152923 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:52.154667 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:52.154675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:52.154677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:52.154955 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:52.155227 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:52.155228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:52.155228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:52.155341 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:52.155656 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:52.155670 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:52.155825 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 13:12:52.155922 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 13:12:52.156158 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260926 13:12:52.157202 18788 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45963
I20260926 13:12:52.157294 18788 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:52.157474 18788 heartbeater.cc:511] Master 127.0.98.62:45963 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 13:12:52.157876 18463 ts_manager.cc:194] Registered new tserver with Master: c8138574b1d64c24a2565c0008426351 (127.0.98.2:34701)
I20260926 13:12:52.158270 18463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:12:52.175910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45963
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:52.256701 18793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:52.256950 18793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:52.256979 18793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:52.258311 18793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:52.258364 18793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:52.258391 18793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:12:52.260023 18793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:52.260318 18793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:52.260507 18793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:12:52.265552 18793 server_base.cc:1034] running on GCE node
W20260926 13:12:52.265537 18799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:52.265517 18798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:52.265519 18801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:52.266111 18793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:52.266345 18793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:52.267508 18793 hybrid_clock.cc:648] HybridClock initialized: now 1790428372267494 us; error 30 us; skew 500 ppm
Sep 26 13:12:52 dist-test-slave-2dw5 krb5kdc[18418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:52.270195 18793 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:12:52.270661 18793 webserver.cc:460] Webserver started at http://127.0.98.3:37103/ using document root <none> and password file <none>
I20260926 13:12:52.270913 18793 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:52.270967 18793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:52.271086 18793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:52.272147 18793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "9402223229edc699c566284e8991a4d1"
server_key_iv: "3b0e8440f0dd9890ea3d29c3a580cc97"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.272470 18793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "9402223229edc699c566284e8991a4d1"
server_key_iv: "3b0e8440f0dd9890ea3d29c3a580cc97"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.273753 18793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:12:52.274446 18808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:52.274710 18793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:52.274823 18793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "9402223229edc699c566284e8991a4d1"
server_key_iv: "3b0e8440f0dd9890ea3d29c3a580cc97"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.274925 18793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:52.306368 18793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:52.308584 18793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:52.308727 18793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:52.308939 18793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:52.309239 18793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:52.309271 18793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:52.309299 18793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:52.309322 18793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:52.315299 18793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38375
I20260926 13:12:52.315344 18921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38375 every 8 connection(s)
I20260926 13:12:52.315661 18793 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:12:52.320863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 18793
I20260926 13:12:52.320962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:12:52.321153 392 external_mini_cluster.cc:1428] Setting key be28081803c7ecb3ef4c0264a3bb8efb
Sep 26 13:12:52 dist-test-slave-2dw5 krb5kdc[18418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:52.325317 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.316446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48801 (local address 127.0.98.62:45963)
0926 13:12:52.316532 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:52.316534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:52.316691 (+ 157us) server_negotiation.cc:407] Connection header received
0926 13:12:52.317758 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:52.317760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:52.317772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:52.317818 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:52.318474 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.319045 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.320031 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.320229 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.320284 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.322535 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:52.322536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:52.322537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:52.322543 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:52.323916 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:52.324300 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:52.324300 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:52.324301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:52.324352 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:52.324655 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:52.324656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:52.324657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:52.324792 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:52.324859 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:52.325165 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 13:12:52.325238 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:12:52.325407 18924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.316373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45963 (local address 127.0.98.3:48801)
0926 13:12:52.316546 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:52.316600 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:52.317648 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:52.317909 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:52.317913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:52.318187 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:52.318378 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.318382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.319140 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.319141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.319936 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.319938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.320301 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.320302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.320427 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.320964 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:52.320974 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:52.322424 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:52.323996 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:52.324003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:52.324006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:52.324237 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:52.324445 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:52.324446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:52.324446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:52.324573 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:52.324856 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:52.324861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:52.324990 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 13:12:52.325075 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:12:52.325220 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260926 13:12:52.325842 18922 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45963
I20260926 13:12:52.325937 18922 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:52.326179 18922 heartbeater.cc:511] Master 127.0.98.62:45963 requested a full tablet report, sending...
I20260926 13:12:52.326539 18463 ts_manager.cc:194] Registered new tserver with Master: f72fb1ab8a4d40dd9e15df5dcc22ecef (127.0.98.3:38375)
I20260926 13:12:52.327000 18463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48801
I20260926 13:12:52.333133 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:12:52.340987 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.334472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.0.98.62:45963)
0926 13:12:52.334535 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:52.334537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:52.334623 (+ 86us) server_negotiation.cc:407] Connection header received
0926 13:12:52.334682 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:52.334683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:52.334693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:52.334738 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:52.335372 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.335893 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.336748 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.336952 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.337005 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.338350 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:52.338352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:52.338352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:52.338359 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:52.339752 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:52.340184 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:52.340185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:52.340185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:52.340238 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:52.340423 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:52.340424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:52.340424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:52.340562 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:52.340655 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:52.340830 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 13:12:52.340891 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:12:52.342483 18463 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:12:52.344198 18463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:12:52.350386 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.346566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38375 (local address 127.0.0.1:39406)
0926 13:12:52.346769 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:52.346777 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:52.346812 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:52.347166 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:52.347167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:52.347168 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:52.347204 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.347205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.348142 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.348143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.349329 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.349332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.350177 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.350178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.350241 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.350245 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:12:52.350285 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 13:12:52.350317 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
I20260926 13:12:52.350467 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.346588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.0.98.2:34701)
0926 13:12:52.346776 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:52.346781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:52.346791 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:12:52.347121 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:52.347124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:52.347139 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:52.347199 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:52.347415 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.348465 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.349425 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.349986 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.350047 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.350108 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:52.350356 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 13:12:52.350396 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260926 13:12:52.350425 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.346803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.0.98.3:38375)
0926 13:12:52.347007 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:52.347010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:52.347019 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:12:52.347055 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:52.347057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:52.347072 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:52.347266 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:52.347340 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.348046 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.349420 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.350112 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.350181 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.350244 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:52.350298 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 13:12:52.350334 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":130,"threads_started":1}
I20260926 13:12:52.350601 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.346388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34701 (local address 127.0.0.1:33244)
0926 13:12:52.346696 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:52.346706 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:52.347000 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:52.347296 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:52.347297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:52.347299 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:52.347343 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.347345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.348559 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.348560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.349367 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.349369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.350065 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.350066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.350144 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.350158 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 13:12:52.350220 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:12:52.350272 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260926 13:12:52.350811 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.346388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39117 (local address 127.0.0.1:37532)
0926 13:12:52.346545 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:52.346583 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:52.346643 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:52.347061 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:52.347064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:52.347067 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:52.347154 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.347156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.348523 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.348524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.349563 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.349565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.350588 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.350589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.350654 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.350659 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:12:52.350690 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 13:12:52.350742 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":55,"threads_started":1}
I20260926 13:12:52.351250 18856 tablet_service.cc:1467] Processing CreateTablet for tablet fb5e5e56c4e64a4e948afabaf7ff4751 (DEFAULT_TABLE table=test-table [id=28f6f4b14fef4520bc6c46a3068133c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:52.351351 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.346485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.0.98.1:39117)
0926 13:12:52.346892 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:52.346896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:52.346906 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:12:52.346946 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:52.346949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:52.346964 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:52.347048 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:52.347317 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.348419 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.349666 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.350360 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.351144 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.351212 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:52.351238 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 13:12:52.351279 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":78,"threads_started":1}
I20260926 13:12:52.351310 18722 tablet_service.cc:1467] Processing CreateTablet for tablet fb5e5e56c4e64a4e948afabaf7ff4751 (DEFAULT_TABLE table=test-table [id=28f6f4b14fef4520bc6c46a3068133c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:52.351584 18856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5e5e56c4e64a4e948afabaf7ff4751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:52.351635 18722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5e5e56c4e64a4e948afabaf7ff4751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:52.351902 18588 tablet_service.cc:1467] Processing CreateTablet for tablet fb5e5e56c4e64a4e948afabaf7ff4751 (DEFAULT_TABLE table=test-table [id=28f6f4b14fef4520bc6c46a3068133c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:52.352190 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb5e5e56c4e64a4e948afabaf7ff4751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:52.354146 18939 tablet_bootstrap.cc:492] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef: Bootstrap starting.
I20260926 13:12:52.354931 18939 tablet_bootstrap.cc:654] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:52.354978 18940 tablet_bootstrap.cc:492] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351: Bootstrap starting.
I20260926 13:12:52.355309 18941 tablet_bootstrap.cc:492] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac: Bootstrap starting.
I20260926 13:12:52.355612 18940 tablet_bootstrap.cc:654] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:52.355770 18939 log.cc:824] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:52.355844 18941 tablet_bootstrap.cc:654] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:52.356130 18940 log.cc:824] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:52.356359 18941 log.cc:824] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:52.356582 18939 tablet_bootstrap.cc:492] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef: No bootstrap required, opened a new log
I20260926 13:12:52.356652 18939 ts_tablet_manager.cc:1397] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 13:12:52.356808 18940 tablet_bootstrap.cc:492] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351: No bootstrap required, opened a new log
I20260926 13:12:52.356870 18940 ts_tablet_manager.cc:1397] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 13:12:52.356984 18941 tablet_bootstrap.cc:492] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac: No bootstrap required, opened a new log
I20260926 13:12:52.357048 18941 ts_tablet_manager.cc:1397] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:12:52.358605 18941 raft_consensus.cc:348] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39117 } } peers { permanent_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "c8138574b1d64c24a2565c0008426351" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34701 } }
I20260926 13:12:52.358561 18939 raft_consensus.cc:348] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39117 } } peers { permanent_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "c8138574b1d64c24a2565c0008426351" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34701 } }
I20260926 13:12:52.358739 18941 raft_consensus.cc:374] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:52.358741 18939 raft_consensus.cc:374] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:52.358770 18941 raft_consensus.cc:729] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd42ce894a7e417ca6cb1635119bd6ac, State: Initialized, Role: FOLLOWER
I20260926 13:12:52.358777 18939 raft_consensus.cc:729] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f72fb1ab8a4d40dd9e15df5dcc22ecef, State: Initialized, Role: FOLLOWER
I20260926 13:12:52.358752 18940 raft_consensus.cc:348] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39117 } } peers { permanent_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "c8138574b1d64c24a2565c0008426351" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34701 } }
I20260926 13:12:52.358919 18940 raft_consensus.cc:374] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:52.358912 18941 consensus_queue.cc:260] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39117 } } peers { permanent_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "c8138574b1d64c24a2565c0008426351" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34701 } }
I20260926 13:12:52.358959 18940 raft_consensus.cc:729] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8138574b1d64c24a2565c0008426351, State: Initialized, Role: FOLLOWER
I20260926 13:12:52.358999 18939 consensus_queue.cc:260] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39117 } } peers { permanent_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "c8138574b1d64c24a2565c0008426351" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34701 } }
I20260926 13:12:52.359059 18940 consensus_queue.cc:260] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39117 } } peers { permanent_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "c8138574b1d64c24a2565c0008426351" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34701 } }
I20260926 13:12:52.359207 18941 ts_tablet_manager.cc:1428] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:12:52.359263 18939 ts_tablet_manager.cc:1428] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 13:12:52.359287 18940 ts_tablet_manager.cc:1428] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 13:12:52.359333 18654 heartbeater.cc:503] Master 127.0.98.62:45963 was elected leader, sending a full tablet report...
I20260926 13:12:52.359375 18922 heartbeater.cc:503] Master 127.0.98.62:45963 was elected leader, sending a full tablet report...
I20260926 13:12:52.359591 18788 heartbeater.cc:503] Master 127.0.98.62:45963 was elected leader, sending a full tablet report...
W20260926 13:12:52.396584 18789 tablet.cc:2369] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:12:52.481726 18655 tablet.cc:2369] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:12:52.553699 18945 raft_consensus.cc:482] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:52.553843 18945 raft_consensus.cc:504] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39117 } } peers { permanent_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "c8138574b1d64c24a2565c0008426351" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34701 } }
I20260926 13:12:52.554224 18945 leader_election.cc:288] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f72fb1ab8a4d40dd9e15df5dcc22ecef (127.0.98.3:38375), c8138574b1d64c24a2565c0008426351 (127.0.98.2:34701)
I20260926 13:12:52.557220 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.554530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41895 (local address 127.0.98.2:34701)
0926 13:12:52.554579 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:52.554597 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:52.554609 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:12:52.554678 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:52.554679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:52.554692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:52.554749 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:52.555039 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.555512 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.556520 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.556899 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.556961 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.556971 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:52.557112 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 13:12:52.557153 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:12:52.557219 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.554519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40633 (local address 127.0.98.3:38375)
0926 13:12:52.554569 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:52.554571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:52.554714 (+ 143us) server_negotiation.cc:407] Connection header received
0926 13:12:52.554769 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:52.554769 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:52.554777 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:52.554829 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:52.555005 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.555489 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.556534 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.556891 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.556951 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.556963 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:52.557106 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 13:12:52.557142 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 13:12:52.557267 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.554413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34701 (local address 127.0.98.1:41895)
0926 13:12:52.554544 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:52.554557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:52.554615 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:52.554761 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:52.554761 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:52.554763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:52.554885 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.554888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.555594 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.555595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.556434 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.556436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.556977 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.556978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.557041 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.557044 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:52.557093 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 13:12:52.557142 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":6,"thread_start_us":68,"threads_started":1}
I20260926 13:12:52.557274 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.554413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38375 (local address 127.0.98.1:40633)
0926 13:12:52.554637 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:52.554647 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:52.554718 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:52.554867 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:52.554868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:52.554870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:52.554938 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.554940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.555573 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.555574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.556443 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.556446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.556973 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.556975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.557043 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.557046 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:52.557094 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 13:12:52.557144 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":40,"thread_start_us":63,"threads_started":1}
I20260926 13:12:52.557583 18742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5e5e56c4e64a4e948afabaf7ff4751" candidate_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8138574b1d64c24a2565c0008426351" is_pre_election: true
I20260926 13:12:52.557718 18742 raft_consensus.cc:2454] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd42ce894a7e417ca6cb1635119bd6ac in term 0.
I20260926 13:12:52.558069 18542 leader_election.cc:302] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8138574b1d64c24a2565c0008426351, cd42ce894a7e417ca6cb1635119bd6ac; no voters:
I20260926 13:12:52.558055 18876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5e5e56c4e64a4e948afabaf7ff4751" candidate_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef" is_pre_election: true
I20260926 13:12:52.558213 18876 raft_consensus.cc:2454] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd42ce894a7e417ca6cb1635119bd6ac in term 0.
I20260926 13:12:52.558228 18945 raft_consensus.cc:2781] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:12:52.558269 18945 raft_consensus.cc:482] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:12:52.558297 18945 raft_consensus.cc:3037] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:52.559046 18945 raft_consensus.cc:504] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39117 } } peers { permanent_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "c8138574b1d64c24a2565c0008426351" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34701 } }
I20260926 13:12:52.559168 18945 leader_election.cc:288] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [CANDIDATE]: Term 1 election: Requested vote from peers f72fb1ab8a4d40dd9e15df5dcc22ecef (127.0.98.3:38375), c8138574b1d64c24a2565c0008426351 (127.0.98.2:34701)
I20260926 13:12:52.559316 18876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5e5e56c4e64a4e948afabaf7ff4751" candidate_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef"
I20260926 13:12:52.559337 18742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb5e5e56c4e64a4e948afabaf7ff4751" candidate_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8138574b1d64c24a2565c0008426351"
I20260926 13:12:52.559398 18876 raft_consensus.cc:3037] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:52.559406 18742 raft_consensus.cc:3037] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:52.560264 18742 raft_consensus.cc:2454] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd42ce894a7e417ca6cb1635119bd6ac in term 1.
I20260926 13:12:52.560264 18876 raft_consensus.cc:2454] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd42ce894a7e417ca6cb1635119bd6ac in term 1.
I20260926 13:12:52.560427 18542 leader_election.cc:302] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8138574b1d64c24a2565c0008426351, cd42ce894a7e417ca6cb1635119bd6ac; no voters:
I20260926 13:12:52.560519 18945 raft_consensus.cc:2781] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:52.560669 18945 raft_consensus.cc:686] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [term 1 LEADER]: Becoming Leader. State: Replica: cd42ce894a7e417ca6cb1635119bd6ac, State: Running, Role: LEADER
I20260926 13:12:52.560755 18945 consensus_queue.cc:237] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39117 } } peers { permanent_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } } peers { permanent_uuid: "c8138574b1d64c24a2565c0008426351" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34701 } }
I20260926 13:12:52.561563 18463 catalog_manager.cc:5495] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac reported cstate change: term changed from 0 to 1, leader changed from <none> to cd42ce894a7e417ca6cb1635119bd6ac (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd42ce894a7e417ca6cb1635119bd6ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8138574b1d64c24a2565c0008426351" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34701 } health_report { overall_health: UNKNOWN } } }
W20260926 13:12:52.566223 18923 tablet.cc:2369] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:12:52.577466 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.575210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.0.98.1:39117)
0926 13:12:52.575275 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:52.575278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:52.575286 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:12:52.575322 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:52.575323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:52.575344 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:52.575401 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:12:52.575528 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.576021 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.576753 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.576936 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.576993 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.577156 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:12:52.577248 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:12:52.577332 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:52.577378 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 13:12:52.577389 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:12:52.579185 18876 raft_consensus.cc:1261] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef [term 1 FOLLOWER]: Refusing update from remote peer cd42ce894a7e417ca6cb1635119bd6ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:52.579185 18742 raft_consensus.cc:1261] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351 [term 1 FOLLOWER]: Refusing update from remote peer cd42ce894a7e417ca6cb1635119bd6ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:52.579545 18950 consensus_queue.cc:1035] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8138574b1d64c24a2565c0008426351" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:52.579686 18945 consensus_queue.cc:1035] T fb5e5e56c4e64a4e948afabaf7ff4751 P cd42ce894a7e417ca6cb1635119bd6ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f72fb1ab8a4d40dd9e15df5dcc22ecef" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:52.582449 18955 mvcc.cc:205] Tried to move back new op lower bound from 7333594614082039808 to 7333594614009679872. Current Snapshot: MvccSnapshot[applied={T|T < 7333594614082039808}]
I20260926 13:12:52.582984 18954 mvcc.cc:205] Tried to move back new op lower bound from 7333594614082039808 to 7333594614009679872. Current Snapshot: MvccSnapshot[applied={T|T < 7333594614082039808 or (T in {7333594614082039808})}]
I20260926 13:12:52.584401 18461 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:52.584488 18461 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:52.585227 18461 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b2e2d3f96bd4a7a86ab79c912600f2d: Sending DeleteTablet for 3 replicas of tablet fb5e5e56c4e64a4e948afabaf7ff4751
I20260926 13:12:52.585588 18856 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5e5e56c4e64a4e948afabaf7ff4751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39406
I20260926 13:12:52.585807 18962 tablet_replica.cc:330] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef: stopping tablet replica
I20260926 13:12:52.585812 18722 tablet_service.cc:1514] Processing DeleteTablet for tablet fb5e5e56c4e64a4e948afabaf7ff4751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33244
I20260926 13:12:52.585932 18962 raft_consensus.cc:2229] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:52.586042 18962 raft_consensus.cc:2258] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:52.586100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 18525
I20260926 13:12:52.586956 18962 ts_tablet_manager.cc:1905] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:52.588891 18962 ts_tablet_manager.cc:1918] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:52.588963 18962 log.cc:1196] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/fb5e5e56c4e64a4e948afabaf7ff4751
I20260926 13:12:52.589224 18962 ts_tablet_manager.cc:1939] T fb5e5e56c4e64a4e948afabaf7ff4751 P f72fb1ab8a4d40dd9e15df5dcc22ecef: Deleting consensus metadata
I20260926 13:12:52.589604 18450 catalog_manager.cc:4841] TS f72fb1ab8a4d40dd9e15df5dcc22ecef (127.0.98.3:38375): tablet fb5e5e56c4e64a4e948afabaf7ff4751 (table test-table [id=28f6f4b14fef4520bc6c46a3068133c9]) successfully deleted
W20260926 13:12:52.590969 18450 connection.cc:535] client connection to 127.0.98.1:39117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 13:12:52.591056 18450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:12:52.591233 18963 tablet_replica.cc:330] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351: stopping tablet replica
I20260926 13:12:52.591347 18963 raft_consensus.cc:2229] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:52.591480 18963 raft_consensus.cc:2258] T fb5e5e56c4e64a4e948afabaf7ff4751 P c8138574b1d64c24a2565c0008426351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:52.591518 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.591399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39117 (local address 127.0.0.1:37546)
0926 13:12:52.591448 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:52.591486 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 13:12:52.591643 18450 catalog_manager.cc:4584] TS cd42ce894a7e417ca6cb1635119bd6ac (127.0.98.1:39117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb5e5e56c4e64a4e948afabaf7ff4751: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39117: connect: Connection refused (error 111)
I20260926 13:12:52.591683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 18659
I20260926 13:12:52.596405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 18793
I20260926 13:12:52.596487 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.596374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34701 (local address 127.0.0.1:33248)
0926 13:12:52.596427 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:52.596451 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 13:12:52.600302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 18431
2026-09-26T13:12:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1988 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T13:12:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:12:52Z Disabled control of system clock
I20260926 13:12:52.625645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39283
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:12:52.707517 18968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:52.707741 18968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:52.707762 18968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:52.708884 18968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:12:52.708925 18968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:52.708942 18968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:52.708953 18968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:12:52.708963 18968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:12:52.710289 18968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:52.710490 18968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:52.710726 18968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:52.715473 18976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:52.715610 18974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:52.715659 18973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:52.715660 18968 server_base.cc:1034] running on GCE node
I20260926 13:12:52.715965 18968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:52.716239 18968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:52.717379 18968 hybrid_clock.cc:648] HybridClock initialized: now 1790428372717359 us; error 38 us; skew 500 ppm
I20260926 13:12:52.718732 18968 webserver.cc:460] Webserver started at http://127.0.98.62:33833/ using document root <none> and password file <none>
I20260926 13:12:52.718982 18968 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:52.719074 18968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:52.719201 18968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:52.720254 18968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "510fa7aa48024cd491dc089382ae4999"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "cd5ecc7f2145e441802d5174303522cb"
server_key_iv: "341fb16e45a0bfa1bc5e1e60a931f6a9"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.720572 18968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "510fa7aa48024cd491dc089382ae4999"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "cd5ecc7f2145e441802d5174303522cb"
server_key_iv: "341fb16e45a0bfa1bc5e1e60a931f6a9"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.721731 18968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:12:52.722416 18982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:52.722575 18968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:12:52.722664 18968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "510fa7aa48024cd491dc089382ae4999"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "cd5ecc7f2145e441802d5174303522cb"
server_key_iv: "341fb16e45a0bfa1bc5e1e60a931f6a9"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.722731 18968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:52.733819 18968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:52.734135 18968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:52.734261 18968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:52.738118 18968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39283
I20260926 13:12:52.738161 19034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39283 every 8 connection(s)
I20260926 13:12:52.738467 18968 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:12:52.739389 19035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:52.739687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 18968
I20260926 13:12:52.739769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:12:52.739960 392 external_mini_cluster.cc:1428] Setting key e774e6550b6fce6baa077b5e1a1f08e1
I20260926 13:12:52.742503 19035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999: Bootstrap starting.
I20260926 13:12:52.743225 19035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:52.743505 19035 log.cc:824] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:52.744248 19035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999: No bootstrap required, opened a new log
I20260926 13:12:52.746025 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.740458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.0.98.62:39283)
0926 13:12:52.740690 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:52.740700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:52.740732 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:12:52.741254 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:52.741262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:52.741343 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:52.741501 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:52.741696 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.743099 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.743992 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.744589 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.744745 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.745317 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:52.745322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:52.745323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:52.745534 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:52.745576 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:52.745592 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:52.745651 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:52.745706 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 13:12:52.745792 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":110,"threads_started":1}
I20260926 13:12:52.746574 19035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510fa7aa48024cd491dc089382ae4999" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39283 } }
I20260926 13:12:52.746747 19035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:52.746783 19035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510fa7aa48024cd491dc089382ae4999, State: Initialized, Role: FOLLOWER
I20260926 13:12:52.746928 19035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510fa7aa48024cd491dc089382ae4999" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39283 } }
I20260926 13:12:52.746994 19035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:12:52.747012 19035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:12:52.747043 19035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:52.747551 19035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510fa7aa48024cd491dc089382ae4999" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39283 } }
I20260926 13:12:52.747632 19035 leader_election.cc:302] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 510fa7aa48024cd491dc089382ae4999; no voters:
I20260926 13:12:52.747769 19035 leader_election.cc:288] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:12:52.747817 19040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:52.747952 19040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [term 1 LEADER]: Becoming Leader. State: Replica: 510fa7aa48024cd491dc089382ae4999, State: Running, Role: LEADER
I20260926 13:12:52.748027 19035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:12:52.748044 19040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510fa7aa48024cd491dc089382ae4999" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39283 } }
I20260926 13:12:52.748435 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 510fa7aa48024cd491dc089382ae4999. Latest consensus state: current_term: 1 leader_uuid: "510fa7aa48024cd491dc089382ae4999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510fa7aa48024cd491dc089382ae4999" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39283 } } }
I20260926 13:12:52.748533 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:52.748672 19041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "510fa7aa48024cd491dc089382ae4999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510fa7aa48024cd491dc089382ae4999" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39283 } } }
I20260926 13:12:52.748739 19041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999 [sys.catalog]: This master's current role is: LEADER
W20260926 13:12:52.749675 19056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:12:52.749730 19056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:12:52.749773 19049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:12:52.749919 19049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:12:52.751220 19049 catalog_manager.cc:1347] Generated new cluster ID: e79d38b1b3d741c39221928f3d3e7cb9
I20260926 13:12:52.751272 19049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:12:52.762601 19049 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:12:52.763332 19049 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:12:52.772464 19049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999: Generated new TSK 0
I20260926 13:12:52.772681 19049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:12:52.778344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39283
--builtin_ntp_servers=127.0.98.20:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:12:52.859903 19059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:52.860121 19059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:52.860152 19059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:52.861310 19059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:52.861352 19059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:52.861390 19059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:12:52.862764 19059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:52.863034 19059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:52.863309 19059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:52.868321 19064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:52.868413 19067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:52.868321 19065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:52.868682 19059 server_base.cc:1034] running on GCE node
I20260926 13:12:52.868937 19059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:52.869253 19059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:52.870430 19059 hybrid_clock.cc:648] HybridClock initialized: now 1790428372870410 us; error 33 us; skew 500 ppm
I20260926 13:12:52.878149 19059 webserver.cc:460] Webserver started at http://127.0.98.1:40123/ using document root <none> and password file <none>
I20260926 13:12:52.878414 19059 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:52.878468 19059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:52.878577 19059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:52.879766 19059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "40cd7bb1081e4e429468e7ba76d1ea3c"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "604e72ee7e67582b47ab81cec99cf961"
server_key_iv: "e2db4aa77ffee2f4ad4b52c61cc51b68"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.880113 19059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "40cd7bb1081e4e429468e7ba76d1ea3c"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "604e72ee7e67582b47ab81cec99cf961"
server_key_iv: "e2db4aa77ffee2f4ad4b52c61cc51b68"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.881454 19059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:12:52.882243 19073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:52.882426 19059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:12:52.882503 19059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "40cd7bb1081e4e429468e7ba76d1ea3c"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "604e72ee7e67582b47ab81cec99cf961"
server_key_iv: "e2db4aa77ffee2f4ad4b52c61cc51b68"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.882560 19059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:52.894906 19059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:52.895208 19059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:52.895309 19059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:52.895511 19059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:52.895779 19059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:52.895807 19059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:52.895835 19059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:52.895850 19059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:52.902328 19059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42301
I20260926 13:12:52.902393 19186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42301 every 8 connection(s)
I20260926 13:12:52.902729 19059 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:12:52.903087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 19059
I20260926 13:12:52.903188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:12:52.903381 392 external_mini_cluster.cc:1428] Setting key 4a6458c4544d72016d81abe4e3b6d34b
I20260926 13:12:52.904560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39283
--builtin_ntp_servers=127.0.98.20:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:12:52.908324 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.903622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57015 (local address 127.0.98.62:39283)
0926 13:12:52.903668 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:52.903670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:52.904004 (+ 334us) server_negotiation.cc:407] Connection header received
0926 13:12:52.904567 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:52.904568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:52.904580 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:52.904629 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:52.905307 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.905807 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.906775 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.907032 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.907087 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.908021 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:52.908023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:52.908023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:52.908029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:52.908046 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:52.908050 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:52.908092 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:52.908170 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 13:12:52.908237 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:12:52.908556 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:52.903595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39283 (local address 127.0.98.1:57015)
0926 13:12:52.903890 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:52.903929 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:52.904475 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:52.904705 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:52.904710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:52.904721 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:52.905224 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.905229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.905894 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.905895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.906689 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:52.906691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:52.907104 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:52.907105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:52.907242 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:52.907875 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:12:52.907876 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:52.907906 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:12:52.907931 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:52.908099 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:52.908105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:52.908108 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:52.908169 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 13:12:52.908285 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":193,"threads_started":1}
I20260926 13:12:52.909018 19187 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39283
I20260926 13:12:52.909132 19187 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:52.909343 19187 heartbeater.cc:511] Master 127.0.98.62:39283 requested a full tablet report, sending...
I20260926 13:12:52.909864 18999 ts_manager.cc:194] Registered new tserver with Master: 40cd7bb1081e4e429468e7ba76d1ea3c (127.0.98.1:42301)
I20260926 13:12:52.910478 18999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57015
W20260926 13:12:52.985371 19190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:52.985569 19190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:52.985590 19190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:52.986792 19190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:52.986833 19190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:52.986898 19190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:12:52.988267 19190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:52.988495 19190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:52.988771 19190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:52.993616 19195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:52.993652 19198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:52.993652 19196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:52.993904 19190 server_base.cc:1034] running on GCE node
I20260926 13:12:52.994083 19190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:52.994302 19190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:52.995443 19190 hybrid_clock.cc:648] HybridClock initialized: now 1790428372995427 us; error 26 us; skew 500 ppm
I20260926 13:12:52.996629 19190 webserver.cc:460] Webserver started at http://127.0.98.2:43261/ using document root <none> and password file <none>
I20260926 13:12:52.996855 19190 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:52.996904 19190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:52.997035 19190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:52.998072 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "56d04e02f46a4df18912d905f4d802e2"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "4190a65d8c9c2b39bb46e3bb55bb0095"
server_key_iv: "260f73086dfe2dee377cd4c00d74526e"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.998394 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "56d04e02f46a4df18912d905f4d802e2"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "4190a65d8c9c2b39bb46e3bb55bb0095"
server_key_iv: "260f73086dfe2dee377cd4c00d74526e"
server_key_version: "encryptionkey@0"
I20260926 13:12:52.999660 19190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:12:53.000396 19204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:53.000628 19190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:12:53.000710 19190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "56d04e02f46a4df18912d905f4d802e2"
format_stamp: "Formatted at 2026-09-26 13:12:52 on dist-test-slave-2dw5"
server_key: "4190a65d8c9c2b39bb46e3bb55bb0095"
server_key_iv: "260f73086dfe2dee377cd4c00d74526e"
server_key_version: "encryptionkey@0"
I20260926 13:12:53.000770 19190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:53.010470 19190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:53.010761 19190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:53.010915 19190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:53.011127 19190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:53.011430 19190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:53.011462 19190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:53.011490 19190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:53.011514 19190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:53.017160 19190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46679
I20260926 13:12:53.017261 19317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46679 every 8 connection(s)
I20260926 13:12:53.017549 19190 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:12:53.019441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 19190
I20260926 13:12:53.019551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:12:53.019745 392 external_mini_cluster.cc:1428] Setting key 6bba8c77a6b60113916cc9917f912abf
I20260926 13:12:53.020725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39283
--builtin_ntp_servers=127.0.98.20:43845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:12:53.023365 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.018380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34209 (local address 127.0.98.62:39283)
0926 13:12:53.018470 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:53.018473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:53.018738 (+ 265us) server_negotiation.cc:407] Connection header received
0926 13:12:53.019281 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:53.019282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:53.019296 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:53.019340 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:53.020133 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.020784 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.021925 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.022125 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.022192 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.023066 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:53.023068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:53.023068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:53.023073 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:53.023095 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:53.023100 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:53.023144 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:53.023253 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 13:12:53.023283 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:12:53.023581 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.018311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39283 (local address 127.0.98.2:34209)
0926 13:12:53.018514 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:53.018551 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:53.019160 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:53.019422 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:53.019427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:53.019436 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:53.020049 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.020055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.020961 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.020963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.021830 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.021833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.022199 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.022200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.022339 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.022917 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:12:53.022918 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:53.022949 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:12:53.022975 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:53.023180 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:53.023186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:53.023189 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:53.023251 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:12:53.023361 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260926 13:12:53.024147 19318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39283
I20260926 13:12:53.024236 19318 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:53.024418 19318 heartbeater.cc:511] Master 127.0.98.62:39283 requested a full tablet report, sending...
I20260926 13:12:53.024828 18999 ts_manager.cc:194] Registered new tserver with Master: 56d04e02f46a4df18912d905f4d802e2 (127.0.98.2:46679)
I20260926 13:12:53.025202 18999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34209
W20260926 13:12:53.102269 19321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:53.102506 19321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:53.102537 19321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:53.103854 19321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:53.103907 19321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:53.103942 19321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:12:53.105404 19321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:53.105643 19321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:53.105913 19321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:53.110710 19327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:53.110867 19326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:53.110911 19321 server_base.cc:1034] running on GCE node
W20260926 13:12:53.110711 19329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:53.111215 19321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:53.111433 19321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:53.112598 19321 hybrid_clock.cc:648] HybridClock initialized: now 1790428373112578 us; error 31 us; skew 500 ppm
I20260926 13:12:53.113888 19321 webserver.cc:460] Webserver started at http://127.0.98.3:40863/ using document root <none> and password file <none>
I20260926 13:12:53.114094 19321 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:53.114146 19321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:53.114266 19321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:53.115314 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "4bb8b4f7b61848e59ebc6e1af0844891"
format_stamp: "Formatted at 2026-09-26 13:12:53 on dist-test-slave-2dw5"
server_key: "5da9b8733b86dba6a38168650417bc64"
server_key_iv: "c9a9e274d5034db16d25bde29b396acc"
server_key_version: "encryptionkey@0"
I20260926 13:12:53.115648 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4bb8b4f7b61848e59ebc6e1af0844891"
format_stamp: "Formatted at 2026-09-26 13:12:53 on dist-test-slave-2dw5"
server_key: "5da9b8733b86dba6a38168650417bc64"
server_key_iv: "c9a9e274d5034db16d25bde29b396acc"
server_key_version: "encryptionkey@0"
I20260926 13:12:53.116830 19321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:12:53.117601 19335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:53.117759 19321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:53.117852 19321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "4bb8b4f7b61848e59ebc6e1af0844891"
format_stamp: "Formatted at 2026-09-26 13:12:53 on dist-test-slave-2dw5"
server_key: "5da9b8733b86dba6a38168650417bc64"
server_key_iv: "c9a9e274d5034db16d25bde29b396acc"
server_key_version: "encryptionkey@0"
I20260926 13:12:53.117913 19321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:53.161093 19321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:53.161424 19321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:53.161557 19321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:53.161808 19321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:53.162133 19321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:53.162166 19321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:53.162195 19321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:53.162212 19321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:53.167603 19321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42409
I20260926 13:12:53.167662 19448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42409 every 8 connection(s)
I20260926 13:12:53.167987 19321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:12:53.173262 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.168748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49863 (local address 127.0.98.62:39283)
0926 13:12:53.168831 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:53.168833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:53.169060 (+ 227us) server_negotiation.cc:407] Connection header received
0926 13:12:53.169533 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:53.169535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:53.169545 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:53.169696 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:53.170395 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.171029 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.172008 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.172197 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.172261 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.173051 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:53.173052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:53.173052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:53.173056 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:53.173072 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:53.173076 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:53.173120 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:53.173181 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 13:12:53.173202 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:12:53.173446 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.168716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39283 (local address 127.0.98.3:49863)
0926 13:12:53.168918 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:53.168961 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:53.169431 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:53.169756 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:53.169759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:53.169767 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:53.170309 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.170314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.171109 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.171110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.171922 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.171925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.172264 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.172265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.172366 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.172915 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:12:53.172915 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:53.172943 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:12:53.172966 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:53.173127 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:53.173129 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:53.173131 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:12:53.173176 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 13:12:53.173254 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260926 13:12:53.173925 19449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39283
I20260926 13:12:53.174006 19449 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:53.174153 19449 heartbeater.cc:511] Master 127.0.98.62:39283 requested a full tablet report, sending...
I20260926 13:12:53.174551 18999 ts_manager.cc:194] Registered new tserver with Master: 4bb8b4f7b61848e59ebc6e1af0844891 (127.0.98.3:42409)
I20260926 13:12:53.175034 18999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49863
I20260926 13:12:53.175928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 19321
I20260926 13:12:53.176018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:12:53.176162 392 external_mini_cluster.cc:1428] Setting key 7783925911acf18c89ab424f2e3d964e
I20260926 13:12:53.177019 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:12:53.181114 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.178292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.0.98.62:39283)
0926 13:12:53.178347 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:53.178349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:53.178439 (+ 90us) server_negotiation.cc:407] Connection header received
0926 13:12:53.178499 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:53.178500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:53.178509 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:53.178551 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:53.178700 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.179217 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.179943 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.180139 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.180192 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.180847 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:53.180848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:53.180848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:53.180852 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:53.180866 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:53.180870 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:53.180943 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:53.181018 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 13:12:53.181028 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:12:53.182943 18999 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:12:53.184819 18999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:12:53.191998 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.187784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.0.98.1:42301)
0926 13:12:53.187937 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:53.187941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:53.187951 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:12:53.187990 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:53.187992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:53.188007 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:53.188345 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:53.188409 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.189526 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.191118 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.191614 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.191677 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.191772 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:53.191869 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 13:12:53.191906 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260926 13:12:53.191990 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.187386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42301 (local address 127.0.0.1:42158)
0926 13:12:53.187618 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:53.187663 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:53.187737 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:53.188171 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:53.188174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:53.188177 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:53.188262 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.188264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.189642 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.189643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.191038 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.191041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.191691 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.191692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.191770 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.191785 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 13:12:53.191855 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:12:53.191906 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":56,"spinlock_wait_cycles":896,"thread_start_us":91,"threads_started":1}
I20260926 13:12:53.192212 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.187587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46679 (local address 127.0.0.1:50432)
0926 13:12:53.188114 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:53.188122 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:53.188160 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:53.188402 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:53.188403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:53.188404 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:53.188447 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.188449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.189667 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.189668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.190981 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.190983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.191978 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.191979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.192043 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.192046 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:53.192087 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 13:12:53.192127 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":30,"thread_start_us":65,"threads_started":1}
I20260926 13:12:53.192433 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.187776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.0.98.2:46679)
0926 13:12:53.187977 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:53.187981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:53.188209 (+ 228us) server_negotiation.cc:407] Connection header received
0926 13:12:53.188254 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:53.188257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:53.188271 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:53.188336 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:53.188518 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.189583 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.191063 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.191808 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.191871 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.191945 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:53.192313 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 13:12:53.192354 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260926 13:12:53.192852 19121 tablet_service.cc:1467] Processing CreateTablet for tablet c4446606869444389fbdd2819380bcde (DEFAULT_TABLE table=test-table [id=fbcd3bfb56ed47aeba7b7cb94aead403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:53.193159 19121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4446606869444389fbdd2819380bcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:53.194483 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.187386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42409 (local address 127.0.0.1:47226)
0926 13:12:53.187953 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:53.187963 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:53.188027 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:53.190999 (+ 2972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:53.191000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:53.191002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:53.191043 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.191044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.192517 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.192518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.193465 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.193467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.194316 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.194317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.194367 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.194370 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:53.194397 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 13:12:53.194428 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":4,"thread_start_us":67,"threads_started":1}
I20260926 13:12:53.194552 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.187565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.0.98.3:42409)
0926 13:12:53.187756 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:53.187760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:53.188022 (+ 262us) server_negotiation.cc:407] Connection header received
0926 13:12:53.190872 (+ 2850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:53.190875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:53.190893 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:53.191108 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:53.191176 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.192182 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.193532 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.194243 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.194321 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.194407 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:53.194434 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 13:12:53.194472 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260926 13:12:53.195230 19383 tablet_service.cc:1467] Processing CreateTablet for tablet c4446606869444389fbdd2819380bcde (DEFAULT_TABLE table=test-table [id=fbcd3bfb56ed47aeba7b7cb94aead403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:53.195623 19383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4446606869444389fbdd2819380bcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:53.195647 19252 tablet_service.cc:1467] Processing CreateTablet for tablet c4446606869444389fbdd2819380bcde (DEFAULT_TABLE table=test-table [id=fbcd3bfb56ed47aeba7b7cb94aead403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:53.195986 19252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4446606869444389fbdd2819380bcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:53.196738 19463 tablet_bootstrap.cc:492] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c: Bootstrap starting.
I20260926 13:12:53.197350 19463 tablet_bootstrap.cc:654] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:53.197558 19463 log.cc:824] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:53.198132 19463 tablet_bootstrap.cc:492] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c: No bootstrap required, opened a new log
I20260926 13:12:53.198182 19463 ts_tablet_manager.cc:1397] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:53.198797 19465 tablet_bootstrap.cc:492] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891: Bootstrap starting.
I20260926 13:12:53.199443 19465 tablet_bootstrap.cc:654] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:53.199764 19465 log.cc:824] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:53.199903 19466 tablet_bootstrap.cc:492] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2: Bootstrap starting.
I20260926 13:12:53.200524 19466 tablet_bootstrap.cc:654] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:53.200592 19465 tablet_bootstrap.cc:492] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891: No bootstrap required, opened a new log
I20260926 13:12:53.200670 19465 ts_tablet_manager.cc:1397] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:12:53.200587 19463 raft_consensus.cc:348] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42409 } } peers { permanent_uuid: "56d04e02f46a4df18912d905f4d802e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46679 } } peers { permanent_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42301 } }
I20260926 13:12:53.200754 19463 raft_consensus.cc:374] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:53.200794 19463 raft_consensus.cc:729] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40cd7bb1081e4e429468e7ba76d1ea3c, State: Initialized, Role: FOLLOWER
I20260926 13:12:53.200843 19466 log.cc:824] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:53.200904 19463 consensus_queue.cc:260] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42409 } } peers { permanent_uuid: "56d04e02f46a4df18912d905f4d802e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46679 } } peers { permanent_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42301 } }
I20260926 13:12:53.201154 19463 ts_tablet_manager.cc:1428] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:12:53.201201 19187 heartbeater.cc:503] Master 127.0.98.62:39283 was elected leader, sending a full tablet report...
I20260926 13:12:53.201581 19466 tablet_bootstrap.cc:492] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2: No bootstrap required, opened a new log
I20260926 13:12:53.201642 19466 ts_tablet_manager.cc:1397] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:12:53.202960 19465 raft_consensus.cc:348] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42409 } } peers { permanent_uuid: "56d04e02f46a4df18912d905f4d802e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46679 } } peers { permanent_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42301 } }
I20260926 13:12:53.203082 19465 raft_consensus.cc:374] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:53.203101 19465 raft_consensus.cc:729] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb8b4f7b61848e59ebc6e1af0844891, State: Initialized, Role: FOLLOWER
I20260926 13:12:53.203197 19465 consensus_queue.cc:260] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42409 } } peers { permanent_uuid: "56d04e02f46a4df18912d905f4d802e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46679 } } peers { permanent_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42301 } }
I20260926 13:12:53.203469 19449 heartbeater.cc:503] Master 127.0.98.62:39283 was elected leader, sending a full tablet report...
I20260926 13:12:53.203467 19465 ts_tablet_manager.cc:1428] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:12:53.204010 19466 raft_consensus.cc:348] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42409 } } peers { permanent_uuid: "56d04e02f46a4df18912d905f4d802e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46679 } } peers { permanent_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42301 } }
I20260926 13:12:53.204186 19466 raft_consensus.cc:374] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:53.204226 19466 raft_consensus.cc:729] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d04e02f46a4df18912d905f4d802e2, State: Initialized, Role: FOLLOWER
I20260926 13:12:53.204319 19466 consensus_queue.cc:260] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42409 } } peers { permanent_uuid: "56d04e02f46a4df18912d905f4d802e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46679 } } peers { permanent_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42301 } }
I20260926 13:12:53.204579 19466 ts_tablet_manager.cc:1428] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 13:12:53.204650 19318 heartbeater.cc:503] Master 127.0.98.62:39283 was elected leader, sending a full tablet report...
W20260926 13:12:53.268098 19319 tablet.cc:2369] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:12:53.403431 19188 tablet.cc:2369] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:12:53.418669 19450 tablet.cc:2369] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:12:53.476218 19470 raft_consensus.cc:482] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:53.476339 19470 raft_consensus.cc:504] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42409 } } peers { permanent_uuid: "56d04e02f46a4df18912d905f4d802e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46679 } } peers { permanent_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42301 } }
I20260926 13:12:53.476634 19470 leader_election.cc:288] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56d04e02f46a4df18912d905f4d802e2 (127.0.98.2:46679), 40cd7bb1081e4e429468e7ba76d1ea3c (127.0.98.1:42301)
I20260926 13:12:53.480500 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.476902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57121 (local address 127.0.98.1:42301)
0926 13:12:53.476992 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:53.476994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:53.477310 (+ 316us) server_negotiation.cc:407] Connection header received
0926 13:12:53.477339 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:53.477340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:53.477355 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:53.477402 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:53.477531 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.478229 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.479608 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.480151 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.480223 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.480233 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:53.480392 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 13:12:53.480425 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:12:53.480500 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.476951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47657 (local address 127.0.98.2:46679)
0926 13:12:53.477017 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:53.477019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:53.477029 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:12:53.477058 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:53.477059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:53.477073 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:53.477133 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:53.477331 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.478099 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.479607 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.480160 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.480223 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.480233 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:53.480392 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 13:12:53.480425 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:12:53.480609 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.476844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46679 (local address 127.0.98.3:47657)
0926 13:12:53.476898 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:53.476909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:53.476953 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:53.477162 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:53.477163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:53.477166 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:53.477239 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.477242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.478193 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.478194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.479524 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.479527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.480239 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.480240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.480317 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.480324 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:12:53.480381 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 13:12:53.480520 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":13}
I20260926 13:12:53.480615 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.476844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42301 (local address 127.0.98.3:57121)
0926 13:12:53.477189 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:53.477198 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:53.477264 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:53.477408 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:53.477408 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:53.477409 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:53.477462 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.477464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.478314 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.478316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.479524 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:53.479527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.480238 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.480239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:53.480317 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.480325 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 13:12:53.480381 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 13:12:53.480530 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":284,"thread_start_us":103,"threads_started":1}
I20260926 13:12:53.480962 19272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4446606869444389fbdd2819380bcde" candidate_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d04e02f46a4df18912d905f4d802e2" is_pre_election: true
I20260926 13:12:53.481009 19141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4446606869444389fbdd2819380bcde" candidate_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c" is_pre_election: true
I20260926 13:12:53.481097 19272 raft_consensus.cc:2454] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bb8b4f7b61848e59ebc6e1af0844891 in term 0.
I20260926 13:12:53.481140 19141 raft_consensus.cc:2454] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bb8b4f7b61848e59ebc6e1af0844891 in term 0.
I20260926 13:12:53.481353 19339 leader_election.cc:302] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40cd7bb1081e4e429468e7ba76d1ea3c, 4bb8b4f7b61848e59ebc6e1af0844891; no voters:
I20260926 13:12:53.481484 19470 raft_consensus.cc:2781] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:12:53.481539 19470 raft_consensus.cc:482] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:12:53.481561 19470 raft_consensus.cc:3037] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:53.482200 19470 raft_consensus.cc:504] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42409 } } peers { permanent_uuid: "56d04e02f46a4df18912d905f4d802e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46679 } } peers { permanent_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42301 } }
I20260926 13:12:53.482316 19470 leader_election.cc:288] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [CANDIDATE]: Term 1 election: Requested vote from peers 56d04e02f46a4df18912d905f4d802e2 (127.0.98.2:46679), 40cd7bb1081e4e429468e7ba76d1ea3c (127.0.98.1:42301)
I20260926 13:12:53.482447 19141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4446606869444389fbdd2819380bcde" candidate_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c"
I20260926 13:12:53.482515 19141 raft_consensus.cc:3037] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:53.483247 19141 raft_consensus.cc:2454] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bb8b4f7b61848e59ebc6e1af0844891 in term 1.
I20260926 13:12:53.483345 19272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4446606869444389fbdd2819380bcde" candidate_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56d04e02f46a4df18912d905f4d802e2"
I20260926 13:12:53.483407 19339 leader_election.cc:302] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40cd7bb1081e4e429468e7ba76d1ea3c, 4bb8b4f7b61848e59ebc6e1af0844891; no voters:
I20260926 13:12:53.483426 19272 raft_consensus.cc:3037] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:53.483503 19470 raft_consensus.cc:2781] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:53.483640 19470 raft_consensus.cc:686] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 1 LEADER]: Becoming Leader. State: Replica: 4bb8b4f7b61848e59ebc6e1af0844891, State: Running, Role: LEADER
I20260926 13:12:53.483717 19470 consensus_queue.cc:237] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42409 } } peers { permanent_uuid: "56d04e02f46a4df18912d905f4d802e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46679 } } peers { permanent_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42301 } }
I20260926 13:12:53.484325 19272 raft_consensus.cc:2454] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bb8b4f7b61848e59ebc6e1af0844891 in term 1.
I20260926 13:12:53.484589 18999 catalog_manager.cc:5495] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bb8b4f7b61848e59ebc6e1af0844891 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb8b4f7b61848e59ebc6e1af0844891" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56d04e02f46a4df18912d905f4d802e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42301 } health_report { overall_health: UNKNOWN } } }
I20260926 13:12:53.548331 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.545844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.0.98.3:42409)
0926 13:12:53.545924 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:53.545926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:53.545935 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:12:53.545970 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:53.545971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:53.545992 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:53.546060 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:12:53.546212 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.546678 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.547466 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:53.547691 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:53.547745 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:53.547948 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:12:53.548047 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:12:53.548152 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:53.548227 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 13:12:53.548236 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:12:53.550259 19272 raft_consensus.cc:1261] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2 [term 1 FOLLOWER]: Refusing update from remote peer 4bb8b4f7b61848e59ebc6e1af0844891: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:53.550259 19141 raft_consensus.cc:1261] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c [term 1 FOLLOWER]: Refusing update from remote peer 4bb8b4f7b61848e59ebc6e1af0844891: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:53.550629 19470 consensus_queue.cc:1035] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40cd7bb1081e4e429468e7ba76d1ea3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:53.550804 19473 consensus_queue.cc:1035] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56d04e02f46a4df18912d905f4d802e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:53.553373 19478 mvcc.cc:205] Tried to move back new op lower bound from 7333594618059948032 to 7333594617790545920. Current Snapshot: MvccSnapshot[applied={T|T < 7333594618059948032}]
I20260926 13:12:53.555632 18998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:53.555747 18998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:53.556522 18998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 510fa7aa48024cd491dc089382ae4999: Sending DeleteTablet for 3 replicas of tablet c4446606869444389fbdd2819380bcde
I20260926 13:12:53.556949 19121 tablet_service.cc:1514] Processing DeleteTablet for tablet c4446606869444389fbdd2819380bcde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:53 UTC) from {username='slave'} at 127.0.0.1:42158
I20260926 13:12:53.557189 19485 tablet_replica.cc:330] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c: stopping tablet replica
I20260926 13:12:53.557176 19252 tablet_service.cc:1514] Processing DeleteTablet for tablet c4446606869444389fbdd2819380bcde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:53 UTC) from {username='slave'} at 127.0.0.1:50432
I20260926 13:12:53.557304 19485 raft_consensus.cc:2229] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:53.557427 19485 raft_consensus.cc:2258] T c4446606869444389fbdd2819380bcde P 40cd7bb1081e4e429468e7ba76d1ea3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:53.557476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 19059
I20260926 13:12:53.559587 19383 tablet_service.cc:1514] Processing DeleteTablet for tablet c4446606869444389fbdd2819380bcde with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:53 UTC) from {username='slave'} at 127.0.0.1:47226
W20260926 13:12:53.562791 18986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42301 (error 108)
I20260926 13:12:53.562927 19487 tablet_replica.cc:330] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891: stopping tablet replica
I20260926 13:12:53.562975 19486 tablet_replica.cc:330] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2: stopping tablet replica
I20260926 13:12:53.563043 19487 raft_consensus.cc:2229] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:12:53.563113 19486 raft_consensus.cc:2229] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:53.563162 19487 raft_consensus.cc:2258] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:53.563253 19486 raft_consensus.cc:2258] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:53.563553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 19190
I20260926 13:12:53.563617 19487 ts_tablet_manager.cc:1905] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:53.563760 19486 ts_tablet_manager.cc:1905] T c4446606869444389fbdd2819380bcde P 56d04e02f46a4df18912d905f4d802e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:53.565260 19487 ts_tablet_manager.cc:1918] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:53.565328 19487 log.cc:1196] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/c4446606869444389fbdd2819380bcde
I20260926 13:12:53.565567 19487 ts_tablet_manager.cc:1939] T c4446606869444389fbdd2819380bcde P 4bb8b4f7b61848e59ebc6e1af0844891: Deleting consensus metadata
I20260926 13:12:53.566042 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.565898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42301 (local address 127.0.0.1:42168)
0926 13:12:53.565947 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:53.566003 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 13:12:53.566154 18983 catalog_manager.cc:4841] TS 4bb8b4f7b61848e59ebc6e1af0844891 (127.0.98.3:42409): tablet c4446606869444389fbdd2819380bcde (table test-table [id=fbcd3bfb56ed47aeba7b7cb94aead403]) successfully deleted
W20260926 13:12:53.566300 18986 catalog_manager.cc:4584] TS 40cd7bb1081e4e429468e7ba76d1ea3c (127.0.98.1:42301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4446606869444389fbdd2819380bcde: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42301: connect: Connection refused (error 111)
I20260926 13:12:53.568733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 19321
I20260926 13:12:53.568836 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:53.568706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46679 (local address 127.0.0.1:50442)
0926 13:12:53.568771 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:53.568799 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 13:12:53.573189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 18968
2026-09-26T13:12:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (972 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:12:53 dist-test-slave-2dw5 krb5kdc[19491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:12:53 dist-test-slave-2dw5 krb5kdc[19491](info): set up 2 sockets
Sep 26 13:12:53 dist-test-slave-2dw5 krb5kdc[19491](info): commencing operation
krb5kdc: starting...
W20260926 13:12:54.522682 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:12:54 dist-test-slave-2dw5 krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:12:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:12:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:12:54.642197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35965
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:54.722747 19504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:54.723016 19504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:54.723047 19504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:54.724318 19504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:12:54.724375 19504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:54.724395 19504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:54.724411 19504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:12:54.724426 19504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:12:54.726024 19504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:54.726291 19504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:54.726508 19504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:54.731555 19510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:54.731547 19512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:54.731550 19509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:54.731954 19504 server_base.cc:1034] running on GCE node
I20260926 13:12:54.732204 19504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:54.732493 19504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:54.733651 19504 hybrid_clock.cc:648] HybridClock initialized: now 1790428374733636 us; error 37 us; skew 500 ppm
Sep 26 13:12:54 dist-test-slave-2dw5 krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:54.736340 19504 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:12:54.736847 19504 webserver.cc:460] Webserver started at http://127.0.98.62:39551/ using document root <none> and password file <none>
I20260926 13:12:54.737040 19504 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:54.737079 19504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:54.737198 19504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:54.738293 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "1bd09936187d467b9370226f7ddd7193"
format_stamp: "Formatted at 2026-09-26 13:12:54 on dist-test-slave-2dw5"
server_key: "f00e28dc80125ae25b02d1eb01e7fb8b"
server_key_iv: "7328f388a9cc6720a97f6c99dd354c12"
server_key_version: "encryptionkey@0"
I20260926 13:12:54.738616 19504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "1bd09936187d467b9370226f7ddd7193"
format_stamp: "Formatted at 2026-09-26 13:12:54 on dist-test-slave-2dw5"
server_key: "f00e28dc80125ae25b02d1eb01e7fb8b"
server_key_iv: "7328f388a9cc6720a97f6c99dd354c12"
server_key_version: "encryptionkey@0"
I20260926 13:12:54.739959 19504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:12:54.740720 19519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:54.740911 19504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:54.741015 19504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "1bd09936187d467b9370226f7ddd7193"
format_stamp: "Formatted at 2026-09-26 13:12:54 on dist-test-slave-2dw5"
server_key: "f00e28dc80125ae25b02d1eb01e7fb8b"
server_key_iv: "7328f388a9cc6720a97f6c99dd354c12"
server_key_version: "encryptionkey@0"
I20260926 13:12:54.741097 19504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:54.760161 19504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:54.762117 19504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:54.762243 19504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:54.766290 19504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35965
I20260926 13:12:54.766336 19571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35965 every 8 connection(s)
I20260926 13:12:54.766667 19504 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:12:54.767136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 19504
I20260926 13:12:54.767234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:12:54.767431 392 external_mini_cluster.cc:1428] Setting key da2402f6aa3870c87128fbc12bcdd1a1
I20260926 13:12:54.767596 19572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:54.771140 19572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193: Bootstrap starting.
I20260926 13:12:54.771806 19572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:54.772096 19572 log.cc:824] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:54.772787 19572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193: No bootstrap required, opened a new log
Sep 26 13:12:54 dist-test-slave-2dw5 krb5kdc[19491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:54.774972 19572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd09936187d467b9370226f7ddd7193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } }
I20260926 13:12:54.775130 19572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:54.775163 19572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd09936187d467b9370226f7ddd7193, State: Initialized, Role: FOLLOWER
I20260926 13:12:54.775305 19572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd09936187d467b9370226f7ddd7193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } }
I20260926 13:12:54.775381 19572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:12:54.775414 19572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:12:54.775456 19572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:54.776166 19572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd09936187d467b9370226f7ddd7193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } }
I20260926 13:12:54.776283 19572 leader_election.cc:302] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bd09936187d467b9370226f7ddd7193; no voters:
I20260926 13:12:54.776458 19572 leader_election.cc:288] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:12:54.776501 19577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:54.776625 19577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [term 1 LEADER]: Becoming Leader. State: Replica: 1bd09936187d467b9370226f7ddd7193, State: Running, Role: LEADER
I20260926 13:12:54.776757 19572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:12:54.776794 19577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd09936187d467b9370226f7ddd7193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } }
I20260926 13:12:54.777271 19578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bd09936187d467b9370226f7ddd7193. Latest consensus state: current_term: 1 leader_uuid: "1bd09936187d467b9370226f7ddd7193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd09936187d467b9370226f7ddd7193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } } }
I20260926 13:12:54.777383 19578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:54.777603 19586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:12:54.778137 19586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:12:54.779170 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:54.767958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.0.98.62:35965)
0926 13:12:54.768146 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:54.768155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:54.768189 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:12:54.768860 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:54.768870 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:54.768939 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:54.769097 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:54.769712 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:54.771109 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:54.772048 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:54.772549 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:54.772654 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:54.775145 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:54.775150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:54.775151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:54.775163 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:54.777124 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:54.777538 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:54.777542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:54.777542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:54.777644 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:54.777862 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:54.777863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:54.777863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:54.778336 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:54.778492 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:54.778789 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 13:12:54.778965 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260926 13:12:54.779218 19579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bd09936187d467b9370226f7ddd7193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd09936187d467b9370226f7ddd7193" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35965 } } }
I20260926 13:12:54.779294 19579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:54.780715 19586 catalog_manager.cc:1347] Generated new cluster ID: 5263a0c1f3134c4aa30265b081fc22e7
I20260926 13:12:54.780763 19586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:12:54.790377 19586 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:12:54.791124 19586 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:12:54.796545 19586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193: Generated new TSK 0
I20260926 13:12:54.796737 19586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:12:54.825356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:54.907523 19598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:54.907742 19598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:54.907769 19598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:54.909036 19598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:54.909091 19598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:54.909121 19598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:12:54.910753 19598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:54.911042 19598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:54.911273 19598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:54.916504 19604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:54.916523 19603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:54.916565 19598 server_base.cc:1034] running on GCE node
W20260926 13:12:54.916538 19606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:54.916954 19598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:54.917234 19598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:54.918399 19598 hybrid_clock.cc:648] HybridClock initialized: now 1790428374918378 us; error 34 us; skew 500 ppm
Sep 26 13:12:54 dist-test-slave-2dw5 krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:54.921422 19598 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:12:54.921914 19598 webserver.cc:460] Webserver started at http://127.0.98.1:32863/ using document root <none> and password file <none>
I20260926 13:12:54.922133 19598 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:54.922181 19598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:54.922319 19598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:54.923405 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c"
format_stamp: "Formatted at 2026-09-26 13:12:54 on dist-test-slave-2dw5"
server_key: "bd103ba934227de9b55c575a2b68b546"
server_key_iv: "6585645476c8ef5eded956acebc1b15d"
server_key_version: "encryptionkey@0"
I20260926 13:12:54.923748 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c"
format_stamp: "Formatted at 2026-09-26 13:12:54 on dist-test-slave-2dw5"
server_key: "bd103ba934227de9b55c575a2b68b546"
server_key_iv: "6585645476c8ef5eded956acebc1b15d"
server_key_version: "encryptionkey@0"
I20260926 13:12:54.925034 19598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:12:54.925824 19613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:54.926005 19598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:54.926095 19598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c"
format_stamp: "Formatted at 2026-09-26 13:12:54 on dist-test-slave-2dw5"
server_key: "bd103ba934227de9b55c575a2b68b546"
server_key_iv: "6585645476c8ef5eded956acebc1b15d"
server_key_version: "encryptionkey@0"
I20260926 13:12:54.926162 19598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:54.939224 19598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:54.941429 19598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:54.941584 19598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:54.941900 19598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:54.942211 19598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:54.942245 19598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:54.942274 19598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:54.942298 19598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:54.948086 19598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46871
I20260926 13:12:54.948129 19726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46871 every 8 connection(s)
I20260926 13:12:54.948436 19598 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:12:54.950590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 19598
I20260926 13:12:54.950683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:12:54.950906 392 external_mini_cluster.cc:1428] Setting key 973a11831e0857c39f767d7001429f6c
Sep 26 13:12:54 dist-test-slave-2dw5 krb5kdc[19491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:54.958748 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:54.949112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58041 (local address 127.0.98.62:35965)
0926 13:12:54.949168 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:54.949170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:54.949380 (+ 210us) server_negotiation.cc:407] Connection header received
0926 13:12:54.950312 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:54.950314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:54.950326 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:54.950398 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:54.951278 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:54.951862 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:54.953026 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:54.953235 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:54.953287 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:54.955775 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:54.955776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:54.955777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:54.955785 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:54.957271 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:54.957748 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:54.957749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:54.957749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:54.957865 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:54.958135 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:54.958135 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:54.958135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:54.958242 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:54.958291 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:54.958615 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 13:12:54.958680 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:12:54.959045 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:54.949085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35965 (local address 127.0.98.1:58041)
0926 13:12:54.949255 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:54.949296 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:54.950203 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:54.950511 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:54.950514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:54.950901 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:54.951183 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:54.951189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:54.951951 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:54.951952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:54.952929 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:54.952932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:54.953306 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:54.953307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:54.953454 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:54.953989 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:54.953997 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:54.955639 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:54.957363 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:54.957371 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:54.957374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:54.957675 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:54.957931 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:54.957932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:54.957932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:54.958075 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:54.958300 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:54.958306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:54.958488 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 13:12:54.958607 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 13:12:54.958779 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260926 13:12:54.959566 19727 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35965
I20260926 13:12:54.959662 19727 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:54.959846 19727 heartbeater.cc:511] Master 127.0.98.62:35965 requested a full tablet report, sending...
I20260926 13:12:54.960345 19536 ts_manager.cc:194] Registered new tserver with Master: d3727ec8521e4ac6b8dcfa3a27ef373c (127.0.98.1:46871)
I20260926 13:12:54.961095 19536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:12:54.978500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:55.060458 19732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:55.060678 19732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:55.060709 19732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:55.061957 19732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:55.062007 19732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:55.062043 19732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:12:55.063589 19732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:55.063833 19732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:55.064046 19732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:55.069060 19740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:55.069083 19738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:55.069105 19732 server_base.cc:1034] running on GCE node
W20260926 13:12:55.069072 19737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:55.069532 19732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:55.069747 19732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:55.070901 19732 hybrid_clock.cc:648] HybridClock initialized: now 1790428375070885 us; error 34 us; skew 500 ppm
Sep 26 13:12:55 dist-test-slave-2dw5 krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:55.073459 19732 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:12:55.073938 19732 webserver.cc:460] Webserver started at http://127.0.98.2:38809/ using document root <none> and password file <none>
I20260926 13:12:55.074155 19732 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:55.074203 19732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:55.074322 19732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:55.075423 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "44f58c3955724fa0843bc79329a8154e"
format_stamp: "Formatted at 2026-09-26 13:12:55 on dist-test-slave-2dw5"
server_key: "4d1811d6fce449d64f3add0be838bb9f"
server_key_iv: "da70b5e3879ea110c813180dd4d12f08"
server_key_version: "encryptionkey@0"
I20260926 13:12:55.075748 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "44f58c3955724fa0843bc79329a8154e"
format_stamp: "Formatted at 2026-09-26 13:12:55 on dist-test-slave-2dw5"
server_key: "4d1811d6fce449d64f3add0be838bb9f"
server_key_iv: "da70b5e3879ea110c813180dd4d12f08"
server_key_version: "encryptionkey@0"
I20260926 13:12:55.077065 19732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:12:55.077854 19747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:55.078003 19732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:12:55.078074 19732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "44f58c3955724fa0843bc79329a8154e"
format_stamp: "Formatted at 2026-09-26 13:12:55 on dist-test-slave-2dw5"
server_key: "4d1811d6fce449d64f3add0be838bb9f"
server_key_iv: "da70b5e3879ea110c813180dd4d12f08"
server_key_version: "encryptionkey@0"
I20260926 13:12:55.078145 19732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:55.098106 19732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:55.100618 19732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:55.100775 19732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:55.101017 19732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:55.101358 19732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:55.101395 19732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:55.101424 19732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:55.101442 19732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:55.107146 19732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33487
I20260926 13:12:55.107223 19860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33487 every 8 connection(s)
I20260926 13:12:55.107519 19732 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:12:55.113317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 19732
I20260926 13:12:55.113423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:12:55.113593 392 external_mini_cluster.cc:1428] Setting key 67323bfcd6ce63fc6510f721c21291b5
Sep 26 13:12:55 dist-test-slave-2dw5 krb5kdc[19491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:55.116894 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.108313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50459 (local address 127.0.98.62:35965)
0926 13:12:55.108378 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:55.108381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:55.108577 (+ 196us) server_negotiation.cc:407] Connection header received
0926 13:12:55.109574 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:55.109575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:55.109586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:55.109727 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:55.110367 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.110973 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.111969 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.112161 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.112220 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.114314 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:55.114316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:55.114316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:55.114322 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:55.115649 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:55.116026 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:55.116027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:55.116028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:55.116079 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:55.116302 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:55.116302 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:55.116303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:55.116410 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:55.116464 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:55.116727 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 13:12:55.116803 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:12:55.117070 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.108279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35965 (local address 127.0.98.2:50459)
0926 13:12:55.108447 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:55.108487 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:55.109451 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:55.109785 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:55.109789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:55.110064 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:55.110280 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.110285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.111061 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.111062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.111891 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.111893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.112225 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.112226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.112318 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.112840 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:55.112848 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:55.114221 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:55.115732 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:55.115740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:55.115742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:55.115960 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:55.116133 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:55.116134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:55.116135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:55.116249 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:55.116463 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:55.116470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:55.116608 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 13:12:55.116693 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:12:55.116851 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260926 13:12:55.117561 19861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35965
I20260926 13:12:55.117647 19861 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:55.117823 19861 heartbeater.cc:511] Master 127.0.98.62:35965 requested a full tablet report, sending...
I20260926 13:12:55.118228 19536 ts_manager.cc:194] Registered new tserver with Master: 44f58c3955724fa0843bc79329a8154e (127.0.98.2:33487)
I20260926 13:12:55.118757 19536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50459
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:12:55.139178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:55.219110 19866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:55.219324 19866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:55.219349 19866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:55.220566 19866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:55.220631 19866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:55.220660 19866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:12:55.222252 19866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:55.222503 19866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:55.222731 19866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:55.227906 19872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:55.227932 19871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:55.227906 19874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:55.228188 19866 server_base.cc:1034] running on GCE node
I20260926 13:12:55.228423 19866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:55.228652 19866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:55.229795 19866 hybrid_clock.cc:648] HybridClock initialized: now 1790428375229782 us; error 32 us; skew 500 ppm
Sep 26 13:12:55 dist-test-slave-2dw5 krb5kdc[19491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:55.232466 19866 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:12:55.232946 19866 webserver.cc:460] Webserver started at http://127.0.98.3:35735/ using document root <none> and password file <none>
I20260926 13:12:55.233165 19866 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:55.233212 19866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:55.233354 19866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:55.234506 19866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "238ef1f9f755499e990de9d6b468ca6e"
format_stamp: "Formatted at 2026-09-26 13:12:55 on dist-test-slave-2dw5"
server_key: "086673dc020bf11b83a3d2c4e4d9ac6b"
server_key_iv: "8bf9da726ca79346b888cf7fea8c0307"
server_key_version: "encryptionkey@0"
I20260926 13:12:55.234863 19866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "238ef1f9f755499e990de9d6b468ca6e"
format_stamp: "Formatted at 2026-09-26 13:12:55 on dist-test-slave-2dw5"
server_key: "086673dc020bf11b83a3d2c4e4d9ac6b"
server_key_iv: "8bf9da726ca79346b888cf7fea8c0307"
server_key_version: "encryptionkey@0"
I20260926 13:12:55.236204 19866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:55.237046 19881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:55.237255 19866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:55.237341 19866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "238ef1f9f755499e990de9d6b468ca6e"
format_stamp: "Formatted at 2026-09-26 13:12:55 on dist-test-slave-2dw5"
server_key: "086673dc020bf11b83a3d2c4e4d9ac6b"
server_key_iv: "8bf9da726ca79346b888cf7fea8c0307"
server_key_version: "encryptionkey@0"
I20260926 13:12:55.237411 19866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:55.260234 19866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:55.262490 19866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:55.262645 19866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:55.262979 19866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:55.263327 19866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:55.263363 19866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:55.263392 19866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:55.263415 19866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:55.269021 19866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43553
I20260926 13:12:55.269074 19994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43553 every 8 connection(s)
I20260926 13:12:55.269385 19866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:12:55.274369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 19866
I20260926 13:12:55.274471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:12:55.274633 392 external_mini_cluster.cc:1428] Setting key 224c59f62821db31a989f8eecef38641
Sep 26 13:12:55 dist-test-slave-2dw5 krb5kdc[19491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:12:55.278941 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.270167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36943 (local address 127.0.98.62:35965)
0926 13:12:55.270227 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:55.270229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:55.270369 (+ 140us) server_negotiation.cc:407] Connection header received
0926 13:12:55.271311 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:55.271312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:55.271326 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:55.271386 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:55.272022 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.272552 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.273633 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.273837 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.273914 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.276222 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:55.276224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:55.276224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:55.276232 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:55.277455 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:55.277959 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:55.277960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:55.277960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:55.278015 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:55.278286 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:55.278287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:55.278287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:55.278403 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:55.278459 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:55.278736 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 13:12:55.278824 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:12:55.279153 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.270067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35965 (local address 127.0.98.3:36943)
0926 13:12:55.270223 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:55.270267 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:55.271212 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:55.271443 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:55.271446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:55.271731 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:55.271932 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.271936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.272657 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.272658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.273519 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.273522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.273957 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.273958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.274078 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.274684 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:12:55.274693 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:55.276133 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:55.277554 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:55.277562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:55.277565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:55.277886 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:55.278066 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:12:55.278067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:12:55.278067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:12:55.278223 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:12:55.278460 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:55.278466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:55.278612 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 13:12:55.278704 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:12:55.278898 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260926 13:12:55.279609 19995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35965
I20260926 13:12:55.279707 19995 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:55.279894 19995 heartbeater.cc:511] Master 127.0.98.62:35965 requested a full tablet report, sending...
I20260926 13:12:55.280292 19536 ts_manager.cc:194] Registered new tserver with Master: 238ef1f9f755499e990de9d6b468ca6e (127.0.98.3:43553)
I20260926 13:12:55.280663 19536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36943
I20260926 13:12:55.286484 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:12:55.293893 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.287993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.0.98.62:35965)
0926 13:12:55.288077 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:55.288079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:55.288257 (+ 178us) server_negotiation.cc:407] Connection header received
0926 13:12:55.288283 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:55.288284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:55.288294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:55.288339 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:55.288941 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.289445 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.290173 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.290366 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.290422 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.291581 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:55.291582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:55.291583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:55.291590 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:55.292797 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:55.293103 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:55.293105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:55.293105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:55.293155 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:55.293328 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:55.293328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:55.293329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:55.293485 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:55.293593 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:55.293711 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 13:12:55.293799 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:12:55.295739 19536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:12:55.297667 19536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:12:55.304250 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.299918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33487 (local address 127.0.0.1:56618)
0926 13:12:55.300203 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:55.300212 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:55.300262 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:55.300679 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:55.300682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:55.300685 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:55.300775 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.300777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.301819 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.301819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.303024 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.303027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.303911 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.303912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.303997 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.304020 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:12:55.304094 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 13:12:55.304157 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":90,"threads_started":1}
I20260926 13:12:55.304258 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.300199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.0.98.2:33487)
0926 13:12:55.300356 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:55.300360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:55.300370 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:12:55.300403 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:55.300405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:55.300417 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:55.300485 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:55.300918 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.301725 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.303139 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.303829 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.303904 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.303971 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:55.304118 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 13:12:55.304160 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260926 13:12:55.304631 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.299918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43553 (local address 127.0.0.1:60084)
0926 13:12:55.300299 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:55.300307 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:55.300339 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:55.301100 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:55.301101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:55.301104 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:55.301165 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.301167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.302420 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.302421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.303447 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.303450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.304384 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.304385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.304460 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.304465 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:12:55.304514 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 13:12:55.304557 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260926 13:12:55.304699 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.300156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.0.98.3:43553)
0926 13:12:55.300877 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:55.300881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:55.300889 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:12:55.300925 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:55.300927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:55.300940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:55.301012 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:55.301255 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.302325 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.303541 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.304243 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.304304 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.304353 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:55.304551 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 13:12:55.304602 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":76,"threads_started":1}
I20260926 13:12:55.305034 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.299917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46871 (local address 127.0.0.1:59048)
0926 13:12:55.300086 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:55.300132 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:55.300199 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:55.300762 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:55.300762 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:55.300764 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:55.300896 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.300898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.302261 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.302262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.303272 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.303274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.304246 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.304247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.304658 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.304663 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:12:55.304925 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 13:12:55.304971 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":54,"threads_started":1}
I20260926 13:12:55.305066 19795 tablet_service.cc:1467] Processing CreateTablet for tablet 041a37d5e9574f4999816abd0c8f252b (DEFAULT_TABLE table=test-table [id=9b2258a5d93341d8bc2b04bb651b4a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:55.305285 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.300037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.0.98.1:46871)
0926 13:12:55.300603 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:55.300606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:55.300613 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:12:55.300645 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:55.300648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:55.300662 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:55.300736 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:55.300989 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.302145 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.303362 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.304069 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.304140 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.304208 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:55.305155 (+ 947us) server_negotiation.cc:299] Negotiation successful
0926 13:12:55.305212 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":90,"threads_started":1}
I20260926 13:12:55.305341 19929 tablet_service.cc:1467] Processing CreateTablet for tablet 041a37d5e9574f4999816abd0c8f252b (DEFAULT_TABLE table=test-table [id=9b2258a5d93341d8bc2b04bb651b4a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:55.305505 19795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041a37d5e9574f4999816abd0c8f252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:55.305636 19929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041a37d5e9574f4999816abd0c8f252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:55.307731 19661 tablet_service.cc:1467] Processing CreateTablet for tablet 041a37d5e9574f4999816abd0c8f252b (DEFAULT_TABLE table=test-table [id=9b2258a5d93341d8bc2b04bb651b4a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:55.308080 19661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 041a37d5e9574f4999816abd0c8f252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:55.308450 20012 tablet_bootstrap.cc:492] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e: Bootstrap starting.
I20260926 13:12:55.308846 20013 tablet_bootstrap.cc:492] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e: Bootstrap starting.
I20260926 13:12:55.309371 20012 tablet_bootstrap.cc:654] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:55.309454 20013 tablet_bootstrap.cc:654] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:55.310061 20013 log.cc:824] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:55.310491 20012 log.cc:824] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:55.310896 20013 tablet_bootstrap.cc:492] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e: No bootstrap required, opened a new log
I20260926 13:12:55.310977 20013 ts_tablet_manager.cc:1397] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:12:55.310972 20016 tablet_bootstrap.cc:492] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c: Bootstrap starting.
I20260926 13:12:55.311362 20012 tablet_bootstrap.cc:492] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e: No bootstrap required, opened a new log
I20260926 13:12:55.311420 20012 ts_tablet_manager.cc:1397] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:12:55.311635 20016 tablet_bootstrap.cc:654] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:55.312211 20016 log.cc:824] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:55.313002 20016 tablet_bootstrap.cc:492] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c: No bootstrap required, opened a new log
I20260926 13:12:55.313088 20016 ts_tablet_manager.cc:1397] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:12:55.313098 20012 raft_consensus.cc:348] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238ef1f9f755499e990de9d6b468ca6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43553 } } peers { permanent_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46871 } } peers { permanent_uuid: "44f58c3955724fa0843bc79329a8154e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33487 } }
I20260926 13:12:55.313094 20013 raft_consensus.cc:348] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238ef1f9f755499e990de9d6b468ca6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43553 } } peers { permanent_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46871 } } peers { permanent_uuid: "44f58c3955724fa0843bc79329a8154e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33487 } }
I20260926 13:12:55.313267 20012 raft_consensus.cc:374] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:55.313271 20013 raft_consensus.cc:374] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:55.313308 20012 raft_consensus.cc:729] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f58c3955724fa0843bc79329a8154e, State: Initialized, Role: FOLLOWER
I20260926 13:12:55.313308 20013 raft_consensus.cc:729] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238ef1f9f755499e990de9d6b468ca6e, State: Initialized, Role: FOLLOWER
I20260926 13:12:55.313419 20012 consensus_queue.cc:260] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238ef1f9f755499e990de9d6b468ca6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43553 } } peers { permanent_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46871 } } peers { permanent_uuid: "44f58c3955724fa0843bc79329a8154e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33487 } }
I20260926 13:12:55.313426 20013 consensus_queue.cc:260] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238ef1f9f755499e990de9d6b468ca6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43553 } } peers { permanent_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46871 } } peers { permanent_uuid: "44f58c3955724fa0843bc79329a8154e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33487 } }
I20260926 13:12:55.313689 20012 ts_tablet_manager.cc:1428] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:12:55.313746 20013 ts_tablet_manager.cc:1428] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:12:55.313812 19995 heartbeater.cc:503] Master 127.0.98.62:35965 was elected leader, sending a full tablet report...
I20260926 13:12:55.313835 19861 heartbeater.cc:503] Master 127.0.98.62:35965 was elected leader, sending a full tablet report...
I20260926 13:12:55.315518 20016 raft_consensus.cc:348] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238ef1f9f755499e990de9d6b468ca6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43553 } } peers { permanent_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46871 } } peers { permanent_uuid: "44f58c3955724fa0843bc79329a8154e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33487 } }
I20260926 13:12:55.315672 20016 raft_consensus.cc:374] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:55.315696 20016 raft_consensus.cc:729] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3727ec8521e4ac6b8dcfa3a27ef373c, State: Initialized, Role: FOLLOWER
I20260926 13:12:55.315789 20016 consensus_queue.cc:260] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238ef1f9f755499e990de9d6b468ca6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43553 } } peers { permanent_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46871 } } peers { permanent_uuid: "44f58c3955724fa0843bc79329a8154e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33487 } }
I20260926 13:12:55.316062 20016 ts_tablet_manager.cc:1428] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 13:12:55.316083 19727 heartbeater.cc:503] Master 127.0.98.62:35965 was elected leader, sending a full tablet report...
W20260926 13:12:55.358258 19862 tablet.cc:2369] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:12:55.449234 19728 tablet.cc:2369] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:12:55.519946 19996 tablet.cc:2369] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:12:55.553505 20018 raft_consensus.cc:482] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:55.553648 20018 raft_consensus.cc:504] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238ef1f9f755499e990de9d6b468ca6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43553 } } peers { permanent_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46871 } } peers { permanent_uuid: "44f58c3955724fa0843bc79329a8154e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33487 } }
I20260926 13:12:55.554013 20018 leader_election.cc:288] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 238ef1f9f755499e990de9d6b468ca6e (127.0.98.3:43553), d3727ec8521e4ac6b8dcfa3a27ef373c (127.0.98.1:46871)
I20260926 13:12:55.557108 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.554240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46871 (local address 127.0.98.2:51469)
0926 13:12:55.554290 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:55.554304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:55.554356 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:55.554551 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:55.554552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:55.554554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:55.554627 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.554628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.555395 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.555396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.556272 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.556275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.556839 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.556840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.556915 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.556918 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:55.556963 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 13:12:55.557009 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 13:12:55.557096 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.554360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47005 (local address 127.0.98.3:43553)
0926 13:12:55.554406 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:55.554407 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:55.554694 (+ 287us) server_negotiation.cc:407] Connection header received
0926 13:12:55.554717 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:55.554718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:55.554726 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:55.554760 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:55.554984 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.555494 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.556419 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.556781 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.556844 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.556853 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:55.556989 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 13:12:55.557028 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:12:55.557096 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.554364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51469 (local address 127.0.98.1:46871)
0926 13:12:55.554414 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:55.554416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:55.554426 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:12:55.554457 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:55.554458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:55.554467 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:55.554526 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:55.554698 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.555303 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.556370 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.556758 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.556813 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.556824 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:55.556983 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 13:12:55.557019 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:12:55.557133 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.554240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43553 (local address 127.0.98.2:47005)
0926 13:12:55.554463 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:55.554471 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:55.554521 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:55.554782 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:55.554783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:55.554785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:55.554881 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.554883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.555579 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.555579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.556329 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.556332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.556857 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.556858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.556922 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.556927 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:12:55.556973 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 13:12:55.557013 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":58,"thread_start_us":214,"threads_started":1}
I20260926 13:12:55.557487 19681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041a37d5e9574f4999816abd0c8f252b" candidate_uuid: "44f58c3955724fa0843bc79329a8154e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c" is_pre_election: true
I20260926 13:12:55.557638 19949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041a37d5e9574f4999816abd0c8f252b" candidate_uuid: "44f58c3955724fa0843bc79329a8154e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "238ef1f9f755499e990de9d6b468ca6e" is_pre_election: true
I20260926 13:12:55.557678 19681 raft_consensus.cc:2454] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f58c3955724fa0843bc79329a8154e in term 0.
I20260926 13:12:55.557770 19949 raft_consensus.cc:2454] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f58c3955724fa0843bc79329a8154e in term 0.
I20260926 13:12:55.557934 19748 leader_election.cc:302] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f58c3955724fa0843bc79329a8154e, d3727ec8521e4ac6b8dcfa3a27ef373c; no voters:
I20260926 13:12:55.558070 20018 raft_consensus.cc:2781] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:12:55.558133 20018 raft_consensus.cc:482] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:12:55.558151 20018 raft_consensus.cc:3037] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:55.558799 20018 raft_consensus.cc:504] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238ef1f9f755499e990de9d6b468ca6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43553 } } peers { permanent_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46871 } } peers { permanent_uuid: "44f58c3955724fa0843bc79329a8154e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33487 } }
I20260926 13:12:55.558954 20018 leader_election.cc:288] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [CANDIDATE]: Term 1 election: Requested vote from peers 238ef1f9f755499e990de9d6b468ca6e (127.0.98.3:43553), d3727ec8521e4ac6b8dcfa3a27ef373c (127.0.98.1:46871)
I20260926 13:12:55.559115 19949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041a37d5e9574f4999816abd0c8f252b" candidate_uuid: "44f58c3955724fa0843bc79329a8154e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "238ef1f9f755499e990de9d6b468ca6e"
I20260926 13:12:55.559206 19949 raft_consensus.cc:3037] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:55.559221 19681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "041a37d5e9574f4999816abd0c8f252b" candidate_uuid: "44f58c3955724fa0843bc79329a8154e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c"
I20260926 13:12:55.559298 19681 raft_consensus.cc:3037] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:55.559939 19949 raft_consensus.cc:2454] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f58c3955724fa0843bc79329a8154e in term 1.
I20260926 13:12:55.560014 19681 raft_consensus.cc:2454] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f58c3955724fa0843bc79329a8154e in term 1.
I20260926 13:12:55.560117 19749 leader_election.cc:302] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 238ef1f9f755499e990de9d6b468ca6e, 44f58c3955724fa0843bc79329a8154e; no voters:
I20260926 13:12:55.560200 20018 raft_consensus.cc:2781] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:55.560348 20018 raft_consensus.cc:686] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 1 LEADER]: Becoming Leader. State: Replica: 44f58c3955724fa0843bc79329a8154e, State: Running, Role: LEADER
I20260926 13:12:55.560447 20018 consensus_queue.cc:237] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238ef1f9f755499e990de9d6b468ca6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43553 } } peers { permanent_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46871 } } peers { permanent_uuid: "44f58c3955724fa0843bc79329a8154e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33487 } }
I20260926 13:12:55.561233 19536 catalog_manager.cc:5495] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e reported cstate change: term changed from 0 to 1, leader changed from <none> to 44f58c3955724fa0843bc79329a8154e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "44f58c3955724fa0843bc79329a8154e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238ef1f9f755499e990de9d6b468ca6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44f58c3955724fa0843bc79329a8154e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33487 } health_report { overall_health: HEALTHY } } }
I20260926 13:12:55.589785 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.587275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56634 (local address 127.0.98.2:33487)
0926 13:12:55.587320 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:55.587322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:55.587332 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:12:55.587406 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:55.587407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:55.587426 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:55.587493 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:12:55.587630 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.588128 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.589020 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.589218 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.589274 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.589390 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:12:55.589517 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:12:55.589587 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:55.589698 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 13:12:55.589719 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:12:55.591552 19681 raft_consensus.cc:1261] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c [term 1 FOLLOWER]: Refusing update from remote peer 44f58c3955724fa0843bc79329a8154e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:55.591624 19949 raft_consensus.cc:1261] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e [term 1 FOLLOWER]: Refusing update from remote peer 44f58c3955724fa0843bc79329a8154e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:55.591898 20018 consensus_queue.cc:1035] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3727ec8521e4ac6b8dcfa3a27ef373c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:55.592044 20022 consensus_queue.cc:1035] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [LEADER]: Connected to new peer: Peer: permanent_uuid: "238ef1f9f755499e990de9d6b468ca6e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:55.597209 19536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:55.597329 19536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:55.598215 19536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1bd09936187d467b9370226f7ddd7193: Sending DeleteTablet for 3 replicas of tablet 041a37d5e9574f4999816abd0c8f252b
I20260926 13:12:55.598572 19795 tablet_service.cc:1514] Processing DeleteTablet for tablet 041a37d5e9574f4999816abd0c8f252b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56618
I20260926 13:12:55.598628 19661 tablet_service.cc:1514] Processing DeleteTablet for tablet 041a37d5e9574f4999816abd0c8f252b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59048
I20260926 13:12:55.598680 19929 tablet_service.cc:1514] Processing DeleteTablet for tablet 041a37d5e9574f4999816abd0c8f252b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60084
I20260926 13:12:55.598829 20035 tablet_replica.cc:330] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e: stopping tablet replica
I20260926 13:12:55.598863 20036 tablet_replica.cc:330] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c: stopping tablet replica
I20260926 13:12:55.598963 20035 raft_consensus.cc:2229] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:12:55.598975 20036 raft_consensus.cc:2229] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:55.599081 20035 raft_consensus.cc:2258] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:55.599088 20036 raft_consensus.cc:2258] T 041a37d5e9574f4999816abd0c8f252b P d3727ec8521e4ac6b8dcfa3a27ef373c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:55.599385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 19598
I20260926 13:12:55.600641 20037 tablet_replica.cc:330] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e: stopping tablet replica
I20260926 13:12:55.600764 20037 raft_consensus.cc:2229] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:55.600876 20037 raft_consensus.cc:2258] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:55.600914 20035 ts_tablet_manager.cc:1905] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:55.602720 20035 ts_tablet_manager.cc:1918] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:55.602805 20035 log.cc:1196] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/041a37d5e9574f4999816abd0c8f252b
I20260926 13:12:55.603063 20035 ts_tablet_manager.cc:1939] T 041a37d5e9574f4999816abd0c8f252b P 44f58c3955724fa0843bc79329a8154e: Deleting consensus metadata
I20260926 13:12:55.603519 19523 catalog_manager.cc:4841] TS 44f58c3955724fa0843bc79329a8154e (127.0.98.2:33487): tablet 041a37d5e9574f4999816abd0c8f252b (table test-table [id=9b2258a5d93341d8bc2b04bb651b4a69]) successfully deleted
W20260926 13:12:55.604276 19520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46871 (error 108)
I20260926 13:12:55.604799 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.604655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46871 (local address 127.0.0.1:59054)
0926 13:12:55.604713 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:55.604762 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 13:12:55.604924 19520 catalog_manager.cc:4584] TS d3727ec8521e4ac6b8dcfa3a27ef373c (127.0.98.1:46871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 041a37d5e9574f4999816abd0c8f252b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46871: connect: Connection refused (error 111)
I20260926 13:12:55.605048 20037 ts_tablet_manager.cc:1905] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:55.605369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 19732
I20260926 13:12:55.606667 20037 ts_tablet_manager.cc:1918] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:55.606740 20037 log.cc:1196] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/041a37d5e9574f4999816abd0c8f252b
I20260926 13:12:55.607050 20037 ts_tablet_manager.cc:1939] T 041a37d5e9574f4999816abd0c8f252b P 238ef1f9f755499e990de9d6b468ca6e: Deleting consensus metadata
I20260926 13:12:55.607514 19521 catalog_manager.cc:4841] TS 238ef1f9f755499e990de9d6b468ca6e (127.0.98.3:43553): tablet 041a37d5e9574f4999816abd0c8f252b (table test-table [id=9b2258a5d93341d8bc2b04bb651b4a69]) successfully deleted
I20260926 13:12:55.610118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 19866
I20260926 13:12:55.614216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 19504
2026-09-26T13:12:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2041 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T13:12:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:12:55Z Disabled control of system clock
I20260926 13:12:55.641023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43503
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:12:55.722713 20042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:55.722986 20042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:55.723009 20042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:55.724267 20042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:12:55.724323 20042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:55.724334 20042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:55.724344 20042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:12:55.724355 20042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:12:55.725865 20042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43503
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:55.726135 20042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:55.726430 20042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:55.731326 20047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:55.731320 20048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:55.731560 20042 server_base.cc:1034] running on GCE node
W20260926 13:12:55.731382 20050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:55.731935 20042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:55.732227 20042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:55.733397 20042 hybrid_clock.cc:648] HybridClock initialized: now 1790428375733370 us; error 44 us; skew 500 ppm
I20260926 13:12:55.734779 20042 webserver.cc:460] Webserver started at http://127.0.98.62:39457/ using document root <none> and password file <none>
I20260926 13:12:55.735044 20042 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:55.735103 20042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:55.735231 20042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:55.736277 20042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "640fbd2b475d43d89711ac5ac47feb19"
format_stamp: "Formatted at 2026-09-26 13:12:55 on dist-test-slave-2dw5"
server_key: "c1d15b94a155c0460aad8d2dd307e4c6"
server_key_iv: "195fa6ca07c09b1e9fdce09ed53ba259"
server_key_version: "encryptionkey@0"
I20260926 13:12:55.736621 20042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "640fbd2b475d43d89711ac5ac47feb19"
format_stamp: "Formatted at 2026-09-26 13:12:55 on dist-test-slave-2dw5"
server_key: "c1d15b94a155c0460aad8d2dd307e4c6"
server_key_iv: "195fa6ca07c09b1e9fdce09ed53ba259"
server_key_version: "encryptionkey@0"
I20260926 13:12:55.737916 20042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:12:55.738744 20056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:55.738922 20042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:55.739008 20042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "640fbd2b475d43d89711ac5ac47feb19"
format_stamp: "Formatted at 2026-09-26 13:12:55 on dist-test-slave-2dw5"
server_key: "c1d15b94a155c0460aad8d2dd307e4c6"
server_key_iv: "195fa6ca07c09b1e9fdce09ed53ba259"
server_key_version: "encryptionkey@0"
I20260926 13:12:55.739073 20042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:55.751919 20042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:55.752238 20042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:55.752378 20042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:55.756215 20042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43503
I20260926 13:12:55.756253 20108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43503 every 8 connection(s)
I20260926 13:12:55.756577 20042 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:12:55.757622 20109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:55.760143 20109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19: Bootstrap starting.
I20260926 13:12:55.760787 20109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:55.761051 20109 log.cc:824] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:55.761713 20109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19: No bootstrap required, opened a new log
I20260926 13:12:55.763459 20109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "640fbd2b475d43d89711ac5ac47feb19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43503 } }
I20260926 13:12:55.763690 20109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:55.763751 20109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 640fbd2b475d43d89711ac5ac47feb19, State: Initialized, Role: FOLLOWER
I20260926 13:12:55.763917 20109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "640fbd2b475d43d89711ac5ac47feb19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43503 } }
I20260926 13:12:55.763994 20109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:12:55.764016 20109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:12:55.764061 20109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:55.764595 20109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "640fbd2b475d43d89711ac5ac47feb19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43503 } }
I20260926 13:12:55.764680 20109 leader_election.cc:302] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 640fbd2b475d43d89711ac5ac47feb19; no voters:
I20260926 13:12:55.764883 20109 leader_election.cc:288] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:12:55.764923 20112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:55.765089 20112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [term 1 LEADER]: Becoming Leader. State: Replica: 640fbd2b475d43d89711ac5ac47feb19, State: Running, Role: LEADER
I20260926 13:12:55.765141 20109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:12:55.765197 20112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "640fbd2b475d43d89711ac5ac47feb19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43503 } }
I20260926 13:12:55.765408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 20042
I20260926 13:12:55.765515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:12:55.765718 392 external_mini_cluster.cc:1428] Setting key ebfb71be8b7fea6c2087a707f92dceec
I20260926 13:12:55.765739 20113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "640fbd2b475d43d89711ac5ac47feb19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "640fbd2b475d43d89711ac5ac47feb19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43503 } } }
I20260926 13:12:55.765834 20114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 640fbd2b475d43d89711ac5ac47feb19. Latest consensus state: current_term: 1 leader_uuid: "640fbd2b475d43d89711ac5ac47feb19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "640fbd2b475d43d89711ac5ac47feb19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43503 } } }
I20260926 13:12:55.765880 20114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:55.765879 20113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:55.766089 20116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:12:55.766603 20116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:12:55.768541 20116 catalog_manager.cc:1347] Generated new cluster ID: 28849eb2de134390a4b48158a3477fdd
I20260926 13:12:55.768590 20116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:12:55.772866 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.766655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.0.98.62:43503)
0926 13:12:55.766982 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:55.766992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:55.767026 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:12:55.767260 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:55.767268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:55.767340 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:55.767488 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:55.767760 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.769049 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.770202 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.771077 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.771216 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.772105 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:55.772109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:55.772110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:55.772334 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:55.772380 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:55.772397 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:55.772499 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:55.772538 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 13:12:55.772630 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":70,"threads_started":1}
I20260926 13:12:55.796108 20116 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:12:55.796675 20116 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:12:55.802429 20116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19: Generated new TSK 0
I20260926 13:12:55.802649 20116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:12:55.813139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43503
--builtin_ntp_servers=127.0.98.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:12:55.894634 20133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:55.894908 20133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:55.894937 20133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:55.896148 20133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:55.896193 20133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:55.896219 20133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:12:55.897644 20133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:55.897890 20133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:55.898181 20133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:55.903154 20138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:55.903144 20139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:55.903146 20141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:55.903558 20133 server_base.cc:1034] running on GCE node
I20260926 13:12:55.903786 20133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:55.903998 20133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:55.905157 20133 hybrid_clock.cc:648] HybridClock initialized: now 1790428375905142 us; error 29 us; skew 500 ppm
I20260926 13:12:55.906634 20133 webserver.cc:460] Webserver started at http://127.0.98.1:42949/ using document root <none> and password file <none>
I20260926 13:12:55.906886 20133 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:55.906944 20133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:55.907068 20133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:55.908134 20133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/data/instance:
uuid: "6cc8b84928124fd2b38b3e7471fcf7d7"
format_stamp: "Formatted at 2026-09-26 13:12:55 on dist-test-slave-2dw5"
server_key: "f7f56fccf021f684297dc036bf4705e7"
server_key_iv: "760ef105a4d373f61b483d663f75f757"
server_key_version: "encryptionkey@0"
I20260926 13:12:55.908473 20133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6cc8b84928124fd2b38b3e7471fcf7d7"
format_stamp: "Formatted at 2026-09-26 13:12:55 on dist-test-slave-2dw5"
server_key: "f7f56fccf021f684297dc036bf4705e7"
server_key_iv: "760ef105a4d373f61b483d663f75f757"
server_key_version: "encryptionkey@0"
I20260926 13:12:55.909713 20133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:12:55.910414 20147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:55.910625 20133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:12:55.910707 20133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/wal
uuid: "6cc8b84928124fd2b38b3e7471fcf7d7"
format_stamp: "Formatted at 2026-09-26 13:12:55 on dist-test-slave-2dw5"
server_key: "f7f56fccf021f684297dc036bf4705e7"
server_key_iv: "760ef105a4d373f61b483d663f75f757"
server_key_version: "encryptionkey@0"
I20260926 13:12:55.910775 20133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:55.928140 20133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:55.928458 20133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:55.928584 20133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:55.928795 20133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:55.929106 20133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:55.929138 20133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:55.929168 20133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:55.929191 20133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:55.934759 20133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45717
I20260926 13:12:55.934808 20260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45717 every 8 connection(s)
I20260926 13:12:55.935144 20133 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:12:55.938053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 20133
I20260926 13:12:55.938161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:12:55.938382 392 external_mini_cluster.cc:1428] Setting key dddf45e6da0bdcae0357ea1c956d2fcd
I20260926 13:12:55.939428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43503
--builtin_ntp_servers=127.0.98.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:12:55.941013 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.935948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52089 (local address 127.0.98.62:43503)
0926 13:12:55.936042 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:55.936044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:55.936164 (+ 120us) server_negotiation.cc:407] Connection header received
0926 13:12:55.936671 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:55.936672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:55.936685 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:55.936740 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:55.937553 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.938247 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.939579 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.939789 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.939822 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.940760 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:55.940761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:55.940761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:55.940765 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:55.940781 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:55.940784 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:55.940834 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:55.940929 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 13:12:55.940956 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:12:55.941262 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:55.935864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43503 (local address 127.0.98.1:52089)
0926 13:12:55.936039 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:55.936085 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:55.936576 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:55.936861 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:55.936865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:55.936873 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:55.937450 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.937455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.938337 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.938338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.939481 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:55.939483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:55.939901 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:55.939902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:55.940017 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:55.940587 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:12:55.940588 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:55.940634 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:12:55.940667 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:55.940849 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:55.940853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:55.940856 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:55.940921 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:12:55.941035 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260926 13:12:55.941753 20261 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43503
I20260926 13:12:55.941846 20261 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:55.942044 20261 heartbeater.cc:511] Master 127.0.98.62:43503 requested a full tablet report, sending...
I20260926 13:12:55.942554 20072 ts_manager.cc:194] Registered new tserver with Master: 6cc8b84928124fd2b38b3e7471fcf7d7 (127.0.98.1:45717)
I20260926 13:12:55.943255 20072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52089
W20260926 13:12:56.021297 20264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:56.021536 20264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:56.021567 20264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:56.022775 20264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:56.022826 20264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:56.022889 20264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:12:56.024457 20264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:56.024718 20264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:56.025027 20264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:56.030210 20272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:56.030349 20270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:56.030349 20264 server_base.cc:1034] running on GCE node
W20260926 13:12:56.030498 20269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:56.030707 20264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:56.030967 20264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:56.032130 20264 hybrid_clock.cc:648] HybridClock initialized: now 1790428376032114 us; error 34 us; skew 500 ppm
I20260926 13:12:56.033504 20264 webserver.cc:460] Webserver started at http://127.0.98.2:38331/ using document root <none> and password file <none>
I20260926 13:12:56.033754 20264 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:56.033802 20264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:56.033917 20264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:56.035099 20264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/data/instance:
uuid: "174a5e9028dc47cabb27144299f3d388"
format_stamp: "Formatted at 2026-09-26 13:12:56 on dist-test-slave-2dw5"
server_key: "57e72505c37b88a0f7597398ba62a0dc"
server_key_iv: "83ccbd9d3505c573981529dacb76d11f"
server_key_version: "encryptionkey@0"
I20260926 13:12:56.035426 20264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/wal/instance:
uuid: "174a5e9028dc47cabb27144299f3d388"
format_stamp: "Formatted at 2026-09-26 13:12:56 on dist-test-slave-2dw5"
server_key: "57e72505c37b88a0f7597398ba62a0dc"
server_key_iv: "83ccbd9d3505c573981529dacb76d11f"
server_key_version: "encryptionkey@0"
I20260926 13:12:56.036726 20264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:12:56.037568 20278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:56.037736 20264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:56.037817 20264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/wal
uuid: "174a5e9028dc47cabb27144299f3d388"
format_stamp: "Formatted at 2026-09-26 13:12:56 on dist-test-slave-2dw5"
server_key: "57e72505c37b88a0f7597398ba62a0dc"
server_key_iv: "83ccbd9d3505c573981529dacb76d11f"
server_key_version: "encryptionkey@0"
I20260926 13:12:56.037889 20264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:56.047430 20264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:56.047751 20264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:56.047878 20264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:56.048102 20264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:56.048445 20264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:56.048482 20264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:56.048513 20264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:56.048537 20264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:56.054082 20264 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45521
I20260926 13:12:56.054134 20391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45521 every 8 connection(s)
I20260926 13:12:56.054476 20264 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:12:56.059495 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.055158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36625 (local address 127.0.98.62:43503)
0926 13:12:56.055221 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:56.055223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:56.055433 (+ 210us) server_negotiation.cc:407] Connection header received
0926 13:12:56.055935 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:56.055936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:56.055947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:56.056009 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:56.056653 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.057184 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.058170 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.058369 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.058431 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.059241 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:56.059246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:56.059246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:56.059251 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:56.059265 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:56.059268 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:56.059314 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:56.059407 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 13:12:56.059433 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:12:56.059660 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.055134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43503 (local address 127.0.98.2:36625)
0926 13:12:56.055308 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:56.055350 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:56.055832 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:56.056058 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:56.056061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:56.056068 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:56.056551 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.056555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.057271 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.057272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.058075 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.058077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.058432 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.058433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.058535 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.059098 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:12:56.059099 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:56.059127 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:12:56.059149 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:56.059345 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:56.059348 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:56.059350 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:12:56.059397 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 13:12:56.059479 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260926 13:12:56.060132 20392 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43503
I20260926 13:12:56.060223 20392 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:56.060400 20392 heartbeater.cc:511] Master 127.0.98.62:43503 requested a full tablet report, sending...
I20260926 13:12:56.060794 20072 ts_manager.cc:194] Registered new tserver with Master: 174a5e9028dc47cabb27144299f3d388 (127.0.98.2:45521)
I20260926 13:12:56.061137 20072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36625
I20260926 13:12:56.064651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 20264
I20260926 13:12:56.064751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:12:56.064898 392 external_mini_cluster.cc:1428] Setting key 7dcd0f2fe951a28add7359b290488af6
I20260926 13:12:56.065747 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43503
--builtin_ntp_servers=127.0.98.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:12:56.146705 20395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:56.146970 20395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:56.146996 20395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:56.148182 20395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:56.148226 20395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:56.148250 20395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:12:56.149559 20395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:56.149760 20395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:56.150053 20395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:56.155175 20401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:56.155368 20400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:56.155473 20403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:56.155470 20395 server_base.cc:1034] running on GCE node
I20260926 13:12:56.155725 20395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:56.155941 20395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:56.157085 20395 hybrid_clock.cc:648] HybridClock initialized: now 1790428376157069 us; error 26 us; skew 500 ppm
I20260926 13:12:56.158491 20395 webserver.cc:460] Webserver started at http://127.0.98.3:38137/ using document root <none> and password file <none>
I20260926 13:12:56.158710 20395 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:56.158768 20395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:56.158922 20395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:56.160032 20395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/data/instance:
uuid: "b87c4add83a241e6bdbb49dc7056a178"
format_stamp: "Formatted at 2026-09-26 13:12:56 on dist-test-slave-2dw5"
server_key: "f7b9d5fdb4a7ab4f1b631662dff834a3"
server_key_iv: "dca75859bea86ad7d614965c5c126387"
server_key_version: "encryptionkey@0"
I20260926 13:12:56.160384 20395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b87c4add83a241e6bdbb49dc7056a178"
format_stamp: "Formatted at 2026-09-26 13:12:56 on dist-test-slave-2dw5"
server_key: "f7b9d5fdb4a7ab4f1b631662dff834a3"
server_key_iv: "dca75859bea86ad7d614965c5c126387"
server_key_version: "encryptionkey@0"
I20260926 13:12:56.161620 20395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:12:56.162381 20409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:56.162652 20395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:12:56.162752 20395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/wal
uuid: "b87c4add83a241e6bdbb49dc7056a178"
format_stamp: "Formatted at 2026-09-26 13:12:56 on dist-test-slave-2dw5"
server_key: "f7b9d5fdb4a7ab4f1b631662dff834a3"
server_key_iv: "dca75859bea86ad7d614965c5c126387"
server_key_version: "encryptionkey@0"
I20260926 13:12:56.162832 20395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:56.178712 20395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:56.179064 20395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:56.179205 20395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:56.179435 20395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:12:56.179774 20395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:12:56.179812 20395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:56.179847 20395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:12:56.179872 20395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:56.185940 20395 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40925
I20260926 13:12:56.185981 20522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40925 every 8 connection(s)
I20260926 13:12:56.186312 20395 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:12:56.190558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 20395
I20260926 13:12:56.190670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:12:56.190887 392 external_mini_cluster.cc:1428] Setting key dd93ffd79e8d816531493c48f5d21e89
I20260926 13:12:56.191478 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.187069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46277 (local address 127.0.98.62:43503)
0926 13:12:56.187127 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:56.187129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:56.187405 (+ 276us) server_negotiation.cc:407] Connection header received
0926 13:12:56.187918 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:56.187919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:56.187931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:56.187981 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:56.188630 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.189122 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.190061 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.190274 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.190330 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.191178 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:56.191180 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:56.191180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:56.191185 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:56.191200 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:56.191205 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:56.191250 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:56.191372 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 13:12:56.191405 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:12:56.191694 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.187072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43503 (local address 127.0.98.3:46277)
0926 13:12:56.187275 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:56.187309 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:56.187816 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:56.188049 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:56.188053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:56.188060 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:12:56.188537 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.188541 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.189202 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.189203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.189972 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.189974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.190348 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.190349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.190471 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.191031 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:12:56.191032 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:12:56.191073 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:12:56.191101 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:12:56.191280 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:12:56.191288 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:12:56.191291 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:56.191369 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 13:12:56.191479 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
I20260926 13:12:56.192098 20523 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43503
I20260926 13:12:56.192277 20523 heartbeater.cc:464] Registering TS with master...
I20260926 13:12:56.192426 20523 heartbeater.cc:511] Master 127.0.98.62:43503 requested a full tablet report, sending...
I20260926 13:12:56.192751 20072 ts_manager.cc:194] Registered new tserver with Master: b87c4add83a241e6bdbb49dc7056a178 (127.0.98.3:40925)
I20260926 13:12:56.193109 20072 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46277
I20260926 13:12:56.203039 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:12:56.207180 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.204428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.0.98.62:43503)
0926 13:12:56.204507 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:56.204509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:56.204554 (+ 45us) server_negotiation.cc:407] Connection header received
0926 13:12:56.204607 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:56.204608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:56.204620 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:56.204672 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:56.204807 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.205304 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.206066 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.206282 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.206364 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.206949 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:56.206950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:56.206950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:56.206955 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:56.206969 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:56.206973 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:56.207022 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:56.207063 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 13:12:56.207090 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:12:56.208770 20072 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:12:56.210467 20072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:12:56.216779 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.212857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.0.98.1:45717)
0926 13:12:56.213122 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:56.213127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:56.213136 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:12:56.213176 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:56.213178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:56.213193 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:56.213295 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:56.213566 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.214437 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.215789 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.216398 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.216460 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.216556 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:56.216658 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 13:12:56.216694 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":67,"threads_started":1}
I20260926 13:12:56.216769 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.212759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45717 (local address 127.0.0.1:49470)
0926 13:12:56.212879 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:56.212920 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:56.212986 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:56.213301 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:56.213303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:56.213306 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:56.213374 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.213376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.214507 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.214508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.215703 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.215705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.216475 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.216476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.216551 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.216564 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 13:12:56.216634 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:12:56.216690 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":90,"threads_started":1}
I20260926 13:12:56.217417 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.213001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40925 (local address 127.0.0.1:50858)
0926 13:12:56.213101 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:56.213108 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:56.213144 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:56.213752 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:56.213753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:56.213755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:56.213790 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.213792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.215028 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.215029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.215995 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.215998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.217185 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.217186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.217248 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.217253 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:12:56.217299 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 13:12:56.217341 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":18,"thread_start_us":64,"threads_started":1}
I20260926 13:12:56.217748 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.213332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.0.98.3:40925)
0926 13:12:56.213513 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:56.213516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:56.213525 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:12:56.213561 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:56.213563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:56.213577 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:56.213644 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:56.213886 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.214955 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.216068 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.216750 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.216805 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.217590 (+ 785us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:56.217618 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 13:12:56.217661 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260926 13:12:56.218415 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.212761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45521 (local address 127.0.0.1:45254)
0926 13:12:56.212933 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:56.212943 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:56.212986 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:56.213469 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:56.213470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:56.213472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:56.213519 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.213520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.214915 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.214916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.215944 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.215946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.216833 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.216834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.218259 (+ 1425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.218264 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:12:56.218301 (+ 37us) client_negotiation.cc:240] Negotiation successful
0926 13:12:56.218340 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":22,"thread_start_us":71,"threads_started":1}
I20260926 13:12:56.218413 20457 tablet_service.cc:1467] Processing CreateTablet for tablet 2b37766f966d42bbb26d8651bbcccc46 (DEFAULT_TABLE table=test-table [id=85c7fb295c22404287ae82e9cd799b30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:56.218603 20195 tablet_service.cc:1467] Processing CreateTablet for tablet 2b37766f966d42bbb26d8651bbcccc46 (DEFAULT_TABLE table=test-table [id=85c7fb295c22404287ae82e9cd799b30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:56.218786 20457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b37766f966d42bbb26d8651bbcccc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:56.218816 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.213125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.0.98.2:45521)
0926 13:12:56.213286 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:56.213290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:56.213301 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:12:56.213339 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:56.213341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:56.213355 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:56.213590 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:56.213652 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.214827 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.216012 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.216731 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.218574 (+ 1843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.218664 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:56.218701 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 13:12:56.218742 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260926 13:12:56.218976 20195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b37766f966d42bbb26d8651bbcccc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:56.219430 20326 tablet_service.cc:1467] Processing CreateTablet for tablet 2b37766f966d42bbb26d8651bbcccc46 (DEFAULT_TABLE table=test-table [id=85c7fb295c22404287ae82e9cd799b30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:12:56.219730 20326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b37766f966d42bbb26d8651bbcccc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:56.221578 20537 tablet_bootstrap.cc:492] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7: Bootstrap starting.
I20260926 13:12:56.222029 20538 tablet_bootstrap.cc:492] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178: Bootstrap starting.
I20260926 13:12:56.222345 20537 tablet_bootstrap.cc:654] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:56.222645 20538 tablet_bootstrap.cc:654] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:56.222733 20539 tablet_bootstrap.cc:492] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388: Bootstrap starting.
I20260926 13:12:56.222653 20537 log.cc:824] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:56.223003 20538 log.cc:824] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:56.223312 20539 tablet_bootstrap.cc:654] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:56.223589 20537 tablet_bootstrap.cc:492] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7: No bootstrap required, opened a new log
I20260926 13:12:56.223598 20539 log.cc:824] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:56.223660 20537 ts_tablet_manager.cc:1397] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:12:56.223798 20538 tablet_bootstrap.cc:492] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178: No bootstrap required, opened a new log
I20260926 13:12:56.223858 20538 ts_tablet_manager.cc:1397] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:12:56.224202 20539 tablet_bootstrap.cc:492] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388: No bootstrap required, opened a new log
I20260926 13:12:56.224305 20539 ts_tablet_manager.cc:1397] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:12:56.226142 20537 raft_consensus.cc:348] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45717 } } peers { permanent_uuid: "174a5e9028dc47cabb27144299f3d388" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45521 } } peers { permanent_uuid: "b87c4add83a241e6bdbb49dc7056a178" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } }
I20260926 13:12:56.226336 20537 raft_consensus.cc:374] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:56.226370 20537 raft_consensus.cc:729] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc8b84928124fd2b38b3e7471fcf7d7, State: Initialized, Role: FOLLOWER
I20260926 13:12:56.226424 20538 raft_consensus.cc:348] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45717 } } peers { permanent_uuid: "174a5e9028dc47cabb27144299f3d388" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45521 } } peers { permanent_uuid: "b87c4add83a241e6bdbb49dc7056a178" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } }
I20260926 13:12:56.226554 20538 raft_consensus.cc:374] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:56.226518 20537 consensus_queue.cc:260] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45717 } } peers { permanent_uuid: "174a5e9028dc47cabb27144299f3d388" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45521 } } peers { permanent_uuid: "b87c4add83a241e6bdbb49dc7056a178" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } }
I20260926 13:12:56.226585 20538 raft_consensus.cc:729] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87c4add83a241e6bdbb49dc7056a178, State: Initialized, Role: FOLLOWER
I20260926 13:12:56.226541 20539 raft_consensus.cc:348] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45717 } } peers { permanent_uuid: "174a5e9028dc47cabb27144299f3d388" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45521 } } peers { permanent_uuid: "b87c4add83a241e6bdbb49dc7056a178" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } }
I20260926 13:12:56.226742 20539 raft_consensus.cc:374] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:56.226778 20539 raft_consensus.cc:729] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174a5e9028dc47cabb27144299f3d388, State: Initialized, Role: FOLLOWER
I20260926 13:12:56.226776 20537 ts_tablet_manager.cc:1428] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:12:56.226740 20538 consensus_queue.cc:260] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45717 } } peers { permanent_uuid: "174a5e9028dc47cabb27144299f3d388" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45521 } } peers { permanent_uuid: "b87c4add83a241e6bdbb49dc7056a178" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } }
I20260926 13:12:56.226920 20261 heartbeater.cc:503] Master 127.0.98.62:43503 was elected leader, sending a full tablet report...
I20260926 13:12:56.226931 20539 consensus_queue.cc:260] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45717 } } peers { permanent_uuid: "174a5e9028dc47cabb27144299f3d388" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45521 } } peers { permanent_uuid: "b87c4add83a241e6bdbb49dc7056a178" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } }
I20260926 13:12:56.226989 20538 ts_tablet_manager.cc:1428] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:12:56.227082 20523 heartbeater.cc:503] Master 127.0.98.62:43503 was elected leader, sending a full tablet report...
I20260926 13:12:56.227190 20539 ts_tablet_manager.cc:1428] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:12:56.227167 20545 raft_consensus.cc:482] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:12:56.227298 20392 heartbeater.cc:503] Master 127.0.98.62:43503 was elected leader, sending a full tablet report...
I20260926 13:12:56.227303 20545 raft_consensus.cc:504] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45717 } } peers { permanent_uuid: "174a5e9028dc47cabb27144299f3d388" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45521 } } peers { permanent_uuid: "b87c4add83a241e6bdbb49dc7056a178" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } }
I20260926 13:12:56.227684 20545 leader_election.cc:288] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc8b84928124fd2b38b3e7471fcf7d7 (127.0.98.1:45717), 174a5e9028dc47cabb27144299f3d388 (127.0.98.2:45521)
I20260926 13:12:56.231151 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.228253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44975 (local address 127.0.98.1:45717)
0926 13:12:56.228305 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:56.228307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:56.228314 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:12:56.228335 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:56.228336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:56.228344 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:56.228393 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:56.228549 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.229094 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.230380 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.230756 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.230823 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.230831 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:56.231049 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 13:12:56.231090 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:12:56.231169 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.227857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45717 (local address 127.0.98.3:44975)
0926 13:12:56.228007 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:56.228015 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:56.228053 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:56.228404 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:56.228405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:56.228407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:56.228473 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.228475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.229187 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.229188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.230265 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.230267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.230881 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.230882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.230966 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.230970 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:12:56.231032 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:12:56.231071 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":8,"thread_start_us":94,"threads_started":1}
I20260926 13:12:56.231639 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.227856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45521 (local address 127.0.98.3:43247)
0926 13:12:56.227901 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:56.227912 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:12:56.227946 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:12:56.228255 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:12:56.228256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:12:56.228258 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:12:56.228318 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.228319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.229084 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.229085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.230258 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:12:56.230260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.231397 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.231398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:12:56.231468 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.231471 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:12:56.231527 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 13:12:56.231568 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 13:12:56.231637 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.227893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43247 (local address 127.0.98.2:45521)
0926 13:12:56.228144 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:56.228146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:56.228153 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:12:56.228177 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:56.228178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:56.228186 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:56.228235 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:12:56.228387 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.229010 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.230910 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.231319 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.231373 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.231380 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:56.231539 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 13:12:56.231578 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260926 13:12:56.231776 20215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b37766f966d42bbb26d8651bbcccc46" candidate_uuid: "b87c4add83a241e6bdbb49dc7056a178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7" is_pre_election: true
I20260926 13:12:56.231918 20215 raft_consensus.cc:2454] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87c4add83a241e6bdbb49dc7056a178 in term 0.
I20260926 13:12:56.231918 20346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b37766f966d42bbb26d8651bbcccc46" candidate_uuid: "b87c4add83a241e6bdbb49dc7056a178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174a5e9028dc47cabb27144299f3d388" is_pre_election: true
I20260926 13:12:56.232043 20346 raft_consensus.cc:2454] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b87c4add83a241e6bdbb49dc7056a178 in term 0.
I20260926 13:12:56.232106 20411 leader_election.cc:302] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc8b84928124fd2b38b3e7471fcf7d7, b87c4add83a241e6bdbb49dc7056a178; no voters:
I20260926 13:12:56.232226 20545 raft_consensus.cc:2781] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:12:56.232285 20545 raft_consensus.cc:482] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:12:56.232299 20545 raft_consensus.cc:3037] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:56.232970 20545 raft_consensus.cc:504] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45717 } } peers { permanent_uuid: "174a5e9028dc47cabb27144299f3d388" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45521 } } peers { permanent_uuid: "b87c4add83a241e6bdbb49dc7056a178" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } }
I20260926 13:12:56.233093 20545 leader_election.cc:288] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc8b84928124fd2b38b3e7471fcf7d7 (127.0.98.1:45717), 174a5e9028dc47cabb27144299f3d388 (127.0.98.2:45521)
I20260926 13:12:56.233309 20215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b37766f966d42bbb26d8651bbcccc46" candidate_uuid: "b87c4add83a241e6bdbb49dc7056a178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7"
I20260926 13:12:56.233357 20346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b37766f966d42bbb26d8651bbcccc46" candidate_uuid: "b87c4add83a241e6bdbb49dc7056a178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "174a5e9028dc47cabb27144299f3d388"
I20260926 13:12:56.233387 20215 raft_consensus.cc:3037] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:56.233440 20346 raft_consensus.cc:3037] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:56.234098 20215 raft_consensus.cc:2454] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87c4add83a241e6bdbb49dc7056a178 in term 1.
I20260926 13:12:56.234200 20346 raft_consensus.cc:2454] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b87c4add83a241e6bdbb49dc7056a178 in term 1.
I20260926 13:12:56.234265 20411 leader_election.cc:302] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc8b84928124fd2b38b3e7471fcf7d7, b87c4add83a241e6bdbb49dc7056a178; no voters:
I20260926 13:12:56.234340 20545 raft_consensus.cc:2781] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:56.234396 20545 raft_consensus.cc:686] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 1 LEADER]: Becoming Leader. State: Replica: b87c4add83a241e6bdbb49dc7056a178, State: Running, Role: LEADER
I20260926 13:12:56.234479 20545 consensus_queue.cc:237] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45717 } } peers { permanent_uuid: "174a5e9028dc47cabb27144299f3d388" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45521 } } peers { permanent_uuid: "b87c4add83a241e6bdbb49dc7056a178" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } }
I20260926 13:12:56.235464 20072 catalog_manager.cc:5495] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87c4add83a241e6bdbb49dc7056a178 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b87c4add83a241e6bdbb49dc7056a178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "174a5e9028dc47cabb27144299f3d388" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b87c4add83a241e6bdbb49dc7056a178" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40925 } health_report { overall_health: HEALTHY } } }
I20260926 13:12:56.245097 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.242935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.0.98.3:40925)
0926 13:12:56.242980 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:56.242982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:56.242990 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:12:56.243030 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:56.243031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:56.243051 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:56.243111 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:12:56.243213 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.243684 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.244392 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:56.244593 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:56.244646 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:56.244783 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:12:56.244887 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:12:56.244958 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:56.244977 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 13:12:56.244995 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:12:56.246989 20215 raft_consensus.cc:1261] T 2b37766f966d42bbb26d8651bbcccc46 P 6cc8b84928124fd2b38b3e7471fcf7d7 [term 1 FOLLOWER]: Refusing update from remote peer b87c4add83a241e6bdbb49dc7056a178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:56.247015 20346 raft_consensus.cc:1261] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388 [term 1 FOLLOWER]: Refusing update from remote peer b87c4add83a241e6bdbb49dc7056a178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:12:56.247393 20545 consensus_queue.cc:1035] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "174a5e9028dc47cabb27144299f3d388" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:56.247526 20544 consensus_queue.cc:1035] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc8b84928124fd2b38b3e7471fcf7d7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:12:56.250710 20551 mvcc.cc:205] Tried to move back new op lower bound from 7333594629105164288 to 7333594629057327104. Current Snapshot: MvccSnapshot[applied={T|T < 7333594629105164288}]
I20260926 13:12:56.251056 20552 mvcc.cc:205] Tried to move back new op lower bound from 7333594629105164288 to 7333594629057327104. Current Snapshot: MvccSnapshot[applied={T|T < 7333594629105164288}]
I20260926 13:12:56.251893 20071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:56.251993 20071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:12:56.252704 20071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 640fbd2b475d43d89711ac5ac47feb19: Sending DeleteTablet for 3 replicas of tablet 2b37766f966d42bbb26d8651bbcccc46
I20260926 13:12:56.253026 20326 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b37766f966d42bbb26d8651bbcccc46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:56 UTC) from {username='slave'} at 127.0.0.1:45254
I20260926 13:12:56.253052 20195 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b37766f966d42bbb26d8651bbcccc46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:56 UTC) from {username='slave'} at 127.0.0.1:49470
I20260926 13:12:56.253093 20457 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b37766f966d42bbb26d8651bbcccc46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:12:56 UTC) from {username='slave'} at 127.0.0.1:50858
I20260926 13:12:56.253297 20560 tablet_replica.cc:330] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388: stopping tablet replica
I20260926 13:12:56.253298 20562 tablet_replica.cc:330] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178: stopping tablet replica
I20260926 13:12:56.253391 20562 raft_consensus.cc:2229] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:12:56.253399 20560 raft_consensus.cc:2229] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:12:56.253476 20562 raft_consensus.cc:2258] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:56.253536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 20133
I20260926 13:12:56.253492 20560 raft_consensus.cc:2258] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:12:56.254885 20560 ts_tablet_manager.cc:1905] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:56.255564 20562 ts_tablet_manager.cc:1905] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:12:56.256816 20560 ts_tablet_manager.cc:1918] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:56.256892 20560 log.cc:1196] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-1/wal/wals/2b37766f966d42bbb26d8651bbcccc46
I20260926 13:12:56.257077 20562 ts_tablet_manager.cc:1918] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:12:56.257146 20562 log.cc:1196] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178: Deleting WAL directory at /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428136114168-392-0/minicluster-data/ts-2/wal/wals/2b37766f966d42bbb26d8651bbcccc46
I20260926 13:12:56.257150 20560 ts_tablet_manager.cc:1939] T 2b37766f966d42bbb26d8651bbcccc46 P 174a5e9028dc47cabb27144299f3d388: Deleting consensus metadata
I20260926 13:12:56.257401 20562 ts_tablet_manager.cc:1939] T 2b37766f966d42bbb26d8651bbcccc46 P b87c4add83a241e6bdbb49dc7056a178: Deleting consensus metadata
I20260926 13:12:56.257570 20059 catalog_manager.cc:4841] TS 174a5e9028dc47cabb27144299f3d388 (127.0.98.2:45521): tablet 2b37766f966d42bbb26d8651bbcccc46 (table test-table [id=85c7fb295c22404287ae82e9cd799b30]) successfully deleted
I20260926 13:12:56.258023 20058 catalog_manager.cc:4841] TS b87c4add83a241e6bdbb49dc7056a178 (127.0.98.3:40925): tablet 2b37766f966d42bbb26d8651bbcccc46 (table test-table [id=85c7fb295c22404287ae82e9cd799b30]) successfully deleted
W20260926 13:12:56.259634 20058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45717 (error 108)
I20260926 13:12:56.259852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 20264
I20260926 13:12:56.260072 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:56.259960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45717 (local address 127.0.0.1:49478)
0926 13:12:56.260007 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 13:12:56.260041 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 13:12:56.260174 20058 catalog_manager.cc:4584] TS 6cc8b84928124fd2b38b3e7471fcf7d7 (127.0.98.1:45717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b37766f966d42bbb26d8651bbcccc46: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45717: connect: Connection refused (error 111)
I20260926 13:12:56.265239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 20395
I20260926 13:12:56.270349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 20042
2026-09-26T13:12:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (655 ms)
[----------] 6 tests from EncryptionPolicyTest (8542 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:12:56 dist-test-slave-2dw5 krb5kdc[20566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:12:56 dist-test-slave-2dw5 krb5kdc[20566](info): set up 2 sockets
Sep 26 13:12:56 dist-test-slave-2dw5 krb5kdc[20566](info): commencing operation
krb5kdc: starting...
W20260926 13:12:57.234825 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:12:57 dist-test-slave-2dw5 krb5kdc[20566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:12:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:12:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:12:57.354281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:57.434896 20579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:57.435127 20579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:57.435159 20579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:57.436362 20579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:12:57.436416 20579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:57.436434 20579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:57.436450 20579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:12:57.436465 20579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:12:57.438320 20579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:57.438704 20579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:57.438975 20579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:57.443801 20587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:57.443827 20584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:57.444025 20585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:57.444047 20579 server_base.cc:1034] running on GCE node
I20260926 13:12:57.444276 20579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:57.444571 20579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:57.445720 20579 hybrid_clock.cc:648] HybridClock initialized: now 1790428377445700 us; error 34 us; skew 500 ppm
Sep 26 13:12:57 dist-test-slave-2dw5 krb5kdc[20566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:57.448249 20579 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:12:57.448834 20579 webserver.cc:460] Webserver started at http://127.0.0.1:44213/ using document root <none> and password file <none>
I20260926 13:12:57.449041 20579 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:57.449088 20579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:57.449213 20579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:57.450291 20579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "e784c0bc85fd41b5a38c5bfb7d1835a4"
format_stamp: "Formatted at 2026-09-26 13:12:57 on dist-test-slave-2dw5"
server_key: "cc1b929201992d3c99ae1681e82363ad"
server_key_iv: "24f024a867403e1e0b1e03334065bd1c"
server_key_version: "encryptionkey@0"
I20260926 13:12:57.450626 20579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "e784c0bc85fd41b5a38c5bfb7d1835a4"
format_stamp: "Formatted at 2026-09-26 13:12:57 on dist-test-slave-2dw5"
server_key: "cc1b929201992d3c99ae1681e82363ad"
server_key_iv: "24f024a867403e1e0b1e03334065bd1c"
server_key_version: "encryptionkey@0"
I20260926 13:12:57.452013 20579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 13:12:57.452919 20594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:57.453111 20579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:57.453202 20579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "e784c0bc85fd41b5a38c5bfb7d1835a4"
format_stamp: "Formatted at 2026-09-26 13:12:57 on dist-test-slave-2dw5"
server_key: "cc1b929201992d3c99ae1681e82363ad"
server_key_iv: "24f024a867403e1e0b1e03334065bd1c"
server_key_version: "encryptionkey@0"
I20260926 13:12:57.453284 20579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:57.464318 20579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:57.466528 20579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:57.466665 20579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:57.470517 20579 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37257
I20260926 13:12:57.470604 20646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37257 every 8 connection(s)
I20260926 13:12:57.470966 20579 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:12:57.471829 20647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:57.474321 20647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4: Bootstrap starting.
I20260926 13:12:57.474938 20647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:57.475216 20647 log.cc:824] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:57.476073 20647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4: No bootstrap required, opened a new log
I20260926 13:12:57.477735 20647 raft_consensus.cc:348] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e784c0bc85fd41b5a38c5bfb7d1835a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37257 } }
I20260926 13:12:57.477878 20647 raft_consensus.cc:374] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:57.477902 20647 raft_consensus.cc:729] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e784c0bc85fd41b5a38c5bfb7d1835a4, State: Initialized, Role: FOLLOWER
I20260926 13:12:57.478034 20647 consensus_queue.cc:260] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e784c0bc85fd41b5a38c5bfb7d1835a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37257 } }
I20260926 13:12:57.478102 20647 raft_consensus.cc:388] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:12:57.478132 20647 raft_consensus.cc:482] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:12:57.478163 20647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:57.478683 20647 raft_consensus.cc:504] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e784c0bc85fd41b5a38c5bfb7d1835a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37257 } }
I20260926 13:12:57.478767 20647 leader_election.cc:302] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e784c0bc85fd41b5a38c5bfb7d1835a4; no voters:
I20260926 13:12:57.478952 20647 leader_election.cc:288] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:12:57.478992 20650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:57.479103 20650 raft_consensus.cc:686] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [term 1 LEADER]: Becoming Leader. State: Replica: e784c0bc85fd41b5a38c5bfb7d1835a4, State: Running, Role: LEADER
I20260926 13:12:57.479089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 20579
I20260926 13:12:57.479193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:12:57.479188 20650 consensus_queue.cc:237] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e784c0bc85fd41b5a38c5bfb7d1835a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37257 } }
I20260926 13:12:57.479243 20647 sys_catalog.cc:564] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:12:57.479372 392 external_mini_cluster.cc:1428] Setting key e631b8b82bb30716b3843cabc2094987
I20260926 13:12:57.479490 20650 sys_catalog.cc:455] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e784c0bc85fd41b5a38c5bfb7d1835a4. Latest consensus state: current_term: 1 leader_uuid: "e784c0bc85fd41b5a38c5bfb7d1835a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e784c0bc85fd41b5a38c5bfb7d1835a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37257 } } }
I20260926 13:12:57.479581 20650 sys_catalog.cc:458] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:57.479784 20651 sys_catalog.cc:455] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e784c0bc85fd41b5a38c5bfb7d1835a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e784c0bc85fd41b5a38c5bfb7d1835a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37257 } } }
I20260926 13:12:57.479923 20651 sys_catalog.cc:458] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:57.480095 20655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:12:57.480592 20655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:12:57.482090 20655 catalog_manager.cc:1347] Generated new cluster ID: 09632b3ab99c4e7ebef5f97975dc09f6
I20260926 13:12:57.482139 20655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:12:57 dist-test-slave-2dw5 krb5kdc[20566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:12:57.492009 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:57.479869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60705 (local address 127.0.0.1:37257)
0926 13:12:57.480060 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:57.480069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:57.480775 (+ 706us) server_negotiation.cc:407] Connection header received
0926 13:12:57.481236 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:57.481244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:57.481306 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:57.481456 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:57.482169 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:57.483560 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:57.484636 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:57.485307 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:57.485408 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:57.488304 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:57.488309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:57.488309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:57.488324 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:57.490281 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:57.490703 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:57.490706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:57.490707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:57.490818 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:57.491048 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:57.491049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:57.491049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:57.491383 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:57.491554 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:57.491660 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 13:12:57.491766 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260926 13:12:57.492360 20655 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:12:57.492983 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:12:57.493024 20655 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:12:57.501861 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:57.494470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59945 (local address 127.0.0.1:37257)
0926 13:12:57.494528 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:57.494531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:57.494668 (+ 137us) server_negotiation.cc:407] Connection header received
0926 13:12:57.494728 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:57.494729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:57.494739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:57.494794 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:57.495425 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:57.496269 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:57.497284 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:57.497620 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:57.497692 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:57.499097 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:57.499099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:57.499099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:12:57.499107 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:57.500644 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:57.501048 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:57.501049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:57.501049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:57.501108 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:12:57.501298 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:12:57.501298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:12:57.501299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:12:57.501427 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:57.501548 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:57.501697 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 13:12:57.501762 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:12:57.502120 20655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e784c0bc85fd41b5a38c5bfb7d1835a4: Generated new TSK 0
I20260926 13:12:57.502302 20655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:12:57.505368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 20579
W20260926 13:12:57.509518 20564 connection.cc:535] client connection to 127.0.0.1:37257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37257): Connection reset by peer (error 104)
2026-09-26T13:12:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:12:57 dist-test-slave-2dw5 krb5kdc[20678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:12:57 dist-test-slave-2dw5 krb5kdc[20678](info): set up 2 sockets
Sep 26 13:12:57 dist-test-slave-2dw5 krb5kdc[20678](info): commencing operation
krb5kdc: starting...
W20260926 13:12:58.470710 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:12:58 dist-test-slave-2dw5 krb5kdc[20678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:12:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:12:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:12:58.589764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:58.670889 20691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:58.671100 20691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:58.671123 20691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:58.672358 20691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:12:58.672411 20691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:58.672425 20691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:58.672436 20691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:12:58.672446 20691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:12:58.674029 20691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:58.674276 20691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:58.674499 20691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:58.679481 20696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:58.679493 20691 server_base.cc:1034] running on GCE node
W20260926 13:12:58.679519 20699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:58.679492 20697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:58.679900 20691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:58.680263 20691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:58.681422 20691 hybrid_clock.cc:648] HybridClock initialized: now 1790428378681401 us; error 29 us; skew 500 ppm
Sep 26 13:12:58 dist-test-slave-2dw5 krb5kdc[20678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:58.690215 20691 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:12:58.690649 20691 webserver.cc:460] Webserver started at http://127.0.0.1:45443/ using document root <none> and password file <none>
I20260926 13:12:58.690883 20691 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:58.690940 20691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:58.691035 20691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:58.692037 20691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "c4e1af5d62cf465fab6cd98075c199aa"
format_stamp: "Formatted at 2026-09-26 13:12:58 on dist-test-slave-2dw5"
server_key: "349030932e4d7490affb6b0fa34b541b"
server_key_iv: "9324141fdf70e71a908f01c535911875"
server_key_version: "encryptionkey@0"
I20260926 13:12:58.692312 20691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4e1af5d62cf465fab6cd98075c199aa"
format_stamp: "Formatted at 2026-09-26 13:12:58 on dist-test-slave-2dw5"
server_key: "349030932e4d7490affb6b0fa34b541b"
server_key_iv: "9324141fdf70e71a908f01c535911875"
server_key_version: "encryptionkey@0"
I20260926 13:12:58.693511 20691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:12:58.694396 20706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:58.694555 20691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:12:58.694639 20691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "c4e1af5d62cf465fab6cd98075c199aa"
format_stamp: "Formatted at 2026-09-26 13:12:58 on dist-test-slave-2dw5"
server_key: "349030932e4d7490affb6b0fa34b541b"
server_key_iv: "9324141fdf70e71a908f01c535911875"
server_key_version: "encryptionkey@0"
I20260926 13:12:58.694691 20691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:58.729246 20691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:58.731444 20691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:58.731559 20691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:58.735611 20691 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43093
I20260926 13:12:58.735648 20758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43093 every 8 connection(s)
I20260926 13:12:58.735975 20691 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:12:58.736830 20759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:58.739280 20759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa: Bootstrap starting.
I20260926 13:12:58.739842 20759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:58.740113 20759 log.cc:824] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:58.740872 20759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa: No bootstrap required, opened a new log
I20260926 13:12:58.742564 20759 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e1af5d62cf465fab6cd98075c199aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43093 } }
I20260926 13:12:58.742713 20759 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:58.742734 20759 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e1af5d62cf465fab6cd98075c199aa, State: Initialized, Role: FOLLOWER
I20260926 13:12:58.742900 20759 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e1af5d62cf465fab6cd98075c199aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43093 } }
I20260926 13:12:58.742971 20759 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:12:58.742993 20759 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:12:58.743031 20759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:58.743590 20759 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e1af5d62cf465fab6cd98075c199aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43093 } }
I20260926 13:12:58.743678 20759 leader_election.cc:302] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4e1af5d62cf465fab6cd98075c199aa; no voters:
I20260926 13:12:58.743834 20759 leader_election.cc:288] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:12:58.743880 20762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:58.744102 20759 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:12:58.744108 20762 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [term 1 LEADER]: Becoming Leader. State: Replica: c4e1af5d62cf465fab6cd98075c199aa, State: Running, Role: LEADER
I20260926 13:12:58.744220 20762 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e1af5d62cf465fab6cd98075c199aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43093 } }
I20260926 13:12:58.744568 20764 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4e1af5d62cf465fab6cd98075c199aa. Latest consensus state: current_term: 1 leader_uuid: "c4e1af5d62cf465fab6cd98075c199aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e1af5d62cf465fab6cd98075c199aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43093 } } }
I20260926 13:12:58.744570 20763 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4e1af5d62cf465fab6cd98075c199aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4e1af5d62cf465fab6cd98075c199aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43093 } } }
I20260926 13:12:58.744725 20763 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:58.744696 20764 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:58.744925 20766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:12:58.745124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 20691
I20260926 13:12:58.745235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:12:58.745412 392 external_mini_cluster.cc:1428] Setting key 1eba1ab904675eba85d1412589617e31
I20260926 13:12:58.745671 20766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:12:58.747293 20766 catalog_manager.cc:1347] Generated new cluster ID: c63addd009b045e4b872f3dca1f1b400
I20260926 13:12:58.747391 20766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:12:58.750530 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:58.746031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38615 (local address 127.0.0.1:43093)
0926 13:12:58.746322 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:58.746331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:58.746363 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:12:58.746518 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:58.746525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:58.746579 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:58.746719 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:58.747007 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:58.748003 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:58.748769 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:58.749193 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:58.749283 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:58.749937 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:58.749941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:58.749941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:58.750119 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:58.750150 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:58.750162 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:58.750228 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:58.750330 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 13:12:58.750358 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":64,"threads_started":1}
I20260926 13:12:58.751469 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:12:58.759212 20766 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:12:58.759953 20766 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:12:58.761683 20781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:58.755123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38375 (local address 127.0.0.1:43093)
0926 13:12:58.755163 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:58.755165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:58.758884 (+ 3719us) server_negotiation.cc:407] Connection header received
0926 13:12:58.759001 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:58.759002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:58.759014 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:58.759077 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:58.759244 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:58.759774 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:58.760539 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:12:58.760748 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:12:58.760803 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:12:58.761428 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:58.761430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:58.761430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:58.761435 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:58.761449 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:58.761454 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:58.761498 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:58.761594 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 13:12:58.761606 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:12:58.773622 20766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4e1af5d62cf465fab6cd98075c199aa: Generated new TSK 0
I20260926 13:12:58.773859 20766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:12:58.776584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 20691
W20260926 13:12:58.780440 20676 connection.cc:535] client connection to 127.0.0.1:43093 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43093): Connection reset by peer (error 104)
2026-09-26T13:12:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1270 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:12:58 dist-test-slave-2dw5 krb5kdc[20790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:12:58 dist-test-slave-2dw5 krb5kdc[20790](info): set up 2 sockets
Sep 26 13:12:58 dist-test-slave-2dw5 krb5kdc[20790](info): commencing operation
krb5kdc: starting...
W20260926 13:12:59.738795 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:12:59 dist-test-slave-2dw5 krb5kdc[20790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:12:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:12:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:12:59.857751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:12:59.938000 20803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:12:59.938207 20803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:12:59.938230 20803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:12:59.939427 20803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:12:59.939474 20803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:12:59.939487 20803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:12:59.939499 20803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:12:59.939509 20803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:12:59.940995 20803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:12:59.941227 20803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:12:59.941444 20803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:12:59.946286 20811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:59.946466 20803 server_base.cc:1034] running on GCE node
W20260926 13:12:59.946300 20809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:12:59.946425 20808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:12:59.946830 20803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:12:59.947180 20803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:12:59.948319 20803 hybrid_clock.cc:648] HybridClock initialized: now 1790428379948303 us; error 36 us; skew 500 ppm
Sep 26 13:12:59 dist-test-slave-2dw5 krb5kdc[20790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:12:59.950819 20803 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:12:59.951325 20803 webserver.cc:460] Webserver started at http://127.0.0.1:42337/ using document root <none> and password file <none>
I20260926 13:12:59.951545 20803 fs_manager.cc:359] Metadata directory not provided
I20260926 13:12:59.951591 20803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:12:59.951713 20803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:12:59.952850 20803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "9a03700b8a924de0b04e108ed89efc59"
format_stamp: "Formatted at 2026-09-26 13:12:59 on dist-test-slave-2dw5"
server_key: "7ac66179a568f03aac78048441adc6d7"
server_key_iv: "a2acf3a8ac3f5963da757dac51a85259"
server_key_version: "encryptionkey@0"
I20260926 13:12:59.953187 20803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a03700b8a924de0b04e108ed89efc59"
format_stamp: "Formatted at 2026-09-26 13:12:59 on dist-test-slave-2dw5"
server_key: "7ac66179a568f03aac78048441adc6d7"
server_key_iv: "a2acf3a8ac3f5963da757dac51a85259"
server_key_version: "encryptionkey@0"
I20260926 13:12:59.954473 20803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:12:59.955299 20818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:12:59.955471 20803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:12:59.955554 20803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "9a03700b8a924de0b04e108ed89efc59"
format_stamp: "Formatted at 2026-09-26 13:12:59 on dist-test-slave-2dw5"
server_key: "7ac66179a568f03aac78048441adc6d7"
server_key_iv: "a2acf3a8ac3f5963da757dac51a85259"
server_key_version: "encryptionkey@0"
I20260926 13:12:59.955615 20803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:12:59.958887 20803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:12:59.960902 20803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:12:59.961035 20803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:12:59.964780 20803 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35525
I20260926 13:12:59.964810 20870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35525 every 8 connection(s)
I20260926 13:12:59.965114 20803 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:12:59.966056 20871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:12:59.968276 20871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59: Bootstrap starting.
I20260926 13:12:59.968803 20871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59: Neither blocks nor log segments found. Creating new log.
I20260926 13:12:59.969034 20871 log.cc:824] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59: Log is configured to *not* fsync() on all Append() calls
I20260926 13:12:59.969668 20871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59: No bootstrap required, opened a new log
I20260926 13:12:59.971251 20871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a03700b8a924de0b04e108ed89efc59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260926 13:12:59.971453 20871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:12:59.971496 20871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a03700b8a924de0b04e108ed89efc59, State: Initialized, Role: FOLLOWER
I20260926 13:12:59.971617 20871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a03700b8a924de0b04e108ed89efc59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260926 13:12:59.971678 20871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:12:59.971696 20871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:12:59.971740 20871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:12:59.972259 20871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a03700b8a924de0b04e108ed89efc59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260926 13:12:59.972345 20871 leader_election.cc:302] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a03700b8a924de0b04e108ed89efc59; no voters:
I20260926 13:12:59.972508 20871 leader_election.cc:288] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:12:59.972569 20874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:12:59.972592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 20803
I20260926 13:12:59.972689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:12:59.972759 20871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:12:59.972877 392 external_mini_cluster.cc:1428] Setting key 50ec4b538f42da1086522eae6b87ecfd
I20260926 13:12:59.972985 20874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [term 1 LEADER]: Becoming Leader. State: Replica: 9a03700b8a924de0b04e108ed89efc59, State: Running, Role: LEADER
I20260926 13:12:59.973066 20874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a03700b8a924de0b04e108ed89efc59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } }
I20260926 13:12:59.973877 20876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a03700b8a924de0b04e108ed89efc59. Latest consensus state: current_term: 1 leader_uuid: "9a03700b8a924de0b04e108ed89efc59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a03700b8a924de0b04e108ed89efc59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } } }
I20260926 13:12:59.973958 20876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [sys.catalog]: This master's current role is: LEADER
I20260926 13:12:59.974257 20875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a03700b8a924de0b04e108ed89efc59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a03700b8a924de0b04e108ed89efc59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35525 } } }
I20260926 13:12:59.974311 20875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59 [sys.catalog]: This master's current role is: LEADER
W20260926 13:12:59.975171 20891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:12:59.975224 20891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:12:59.975303 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:59.973545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39247 (local address 127.0.0.1:35525)
0926 13:12:59.973747 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:59.973758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:59.973796 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:12:59.973921 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:59.973928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:59.973971 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:59.974400 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:59.974784 (+ 384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:59.974787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:59.974787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:59.974791 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:59.974829 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:59.974872 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:59.974945 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:59.974985 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 13:12:59.975118 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260926 13:12:59.975311 20885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:12:59.975504 20885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:12:59.976212 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:12:59.976966 20885 catalog_manager.cc:1347] Generated new cluster ID: 511efb08f62c48a3962a59ecdb8879c0
I20260926 13:12:59.977020 20885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:12:59.980012 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:12:59.977719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37885 (local address 127.0.0.1:35525)
0926 13:12:59.977765 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:12:59.977767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:12:59.979088 (+ 1321us) server_negotiation.cc:407] Connection header received
0926 13:12:59.979144 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:12:59.979145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:12:59.979151 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:12:59.979219 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:12:59.979803 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:12:59.979805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:12:59.979805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:12:59.979809 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:12:59.979827 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:12:59.979831 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:12:59.979883 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:12:59.979934 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 13:12:59.979946 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 13:12:59.983397 20885 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:12:59.983623 20885 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:12:59.988682 20885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a03700b8a924de0b04e108ed89efc59: Generated new TSK 0
I20260926 13:12:59.988853 20885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:12:59.992149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 20803
2026-09-26T13:12:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:00 dist-test-slave-2dw5 krb5kdc[20902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:00 dist-test-slave-2dw5 krb5kdc[20902](info): set up 2 sockets
Sep 26 13:13:00 dist-test-slave-2dw5 krb5kdc[20902](info): commencing operation
krb5kdc: starting...
W20260926 13:13:00.942711 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:01 dist-test-slave-2dw5 krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:13:01.062493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:01.142413 20915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:01.142640 20915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:01.142664 20915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:01.143949 20915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:01.144002 20915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:01.144016 20915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:01.144027 20915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:01.144038 20915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:01.145601 20915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:01.145866 20915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:01.146086 20915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:01.151084 20921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:01.151103 20923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:01.151182 20915 server_base.cc:1034] running on GCE node
W20260926 13:13:01.151145 20920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:01.151489 20915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:01.151805 20915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:01.152977 20915 hybrid_clock.cc:648] HybridClock initialized: now 1790428381152944 us; error 45 us; skew 500 ppm
Sep 26 13:13:01 dist-test-slave-2dw5 krb5kdc[20902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:01.155447 20915 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:13:01.155871 20915 webserver.cc:460] Webserver started at http://127.0.0.1:36061/ using document root <none> and password file <none>
I20260926 13:13:01.156051 20915 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:01.156091 20915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:01.156190 20915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:01.157260 20915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "e9621f29237b4ee9ad63d107d41e69e4"
format_stamp: "Formatted at 2026-09-26 13:13:01 on dist-test-slave-2dw5"
server_key: "05b97450c53d3175a6688ff2b7f9c71d"
server_key_iv: "0e053e15d950712732bb839948b63713"
server_key_version: "encryptionkey@0"
I20260926 13:13:01.157562 20915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9621f29237b4ee9ad63d107d41e69e4"
format_stamp: "Formatted at 2026-09-26 13:13:01 on dist-test-slave-2dw5"
server_key: "05b97450c53d3175a6688ff2b7f9c71d"
server_key_iv: "0e053e15d950712732bb839948b63713"
server_key_version: "encryptionkey@0"
I20260926 13:13:01.158766 20915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:13:01.159554 20930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:01.159720 20915 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:13:01.159801 20915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "e9621f29237b4ee9ad63d107d41e69e4"
format_stamp: "Formatted at 2026-09-26 13:13:01 on dist-test-slave-2dw5"
server_key: "05b97450c53d3175a6688ff2b7f9c71d"
server_key_iv: "0e053e15d950712732bb839948b63713"
server_key_version: "encryptionkey@0"
I20260926 13:13:01.159921 20915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:01.168205 20915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:01.170253 20915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:01.170380 20915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:01.174243 20915 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45799
I20260926 13:13:01.174289 20982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45799 every 8 connection(s)
I20260926 13:13:01.174603 20915 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:01.175509 20983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:01.177060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 20915
I20260926 13:13:01.177194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:01.177377 392 external_mini_cluster.cc:1428] Setting key 2f935e7aef171b5f8c42a5d89dd3ed37
I20260926 13:13:01.178356 20983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4: Bootstrap starting.
I20260926 13:13:01.179024 20983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:01.179355 20983 log.cc:824] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:01.180159 20983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4: No bootstrap required, opened a new log
I20260926 13:13:01.182474 20983 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9621f29237b4ee9ad63d107d41e69e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } }
I20260926 13:13:01.182667 20983 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:01.182704 20983 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9621f29237b4ee9ad63d107d41e69e4, State: Initialized, Role: FOLLOWER
I20260926 13:13:01.182894 20983 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9621f29237b4ee9ad63d107d41e69e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } }
I20260926 13:13:01.182997 20983 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:01.183032 20983 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:01.183079 20983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:01.183856 20983 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9621f29237b4ee9ad63d107d41e69e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } }
I20260926 13:13:01.183974 20983 leader_election.cc:302] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9621f29237b4ee9ad63d107d41e69e4; no voters:
I20260926 13:13:01.184156 20983 leader_election.cc:288] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:01.184208 20988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:01.184366 20988 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [term 1 LEADER]: Becoming Leader. State: Replica: e9621f29237b4ee9ad63d107d41e69e4, State: Running, Role: LEADER
I20260926 13:13:01.184427 20983 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:01.184474 20988 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9621f29237b4ee9ad63d107d41e69e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } }
Sep 26 13:13:01 dist-test-slave-2dw5 krb5kdc[20902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:13:01.184824 20988 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9621f29237b4ee9ad63d107d41e69e4. Latest consensus state: current_term: 1 leader_uuid: "e9621f29237b4ee9ad63d107d41e69e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9621f29237b4ee9ad63d107d41e69e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } } }
I20260926 13:13:01.184875 20989 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9621f29237b4ee9ad63d107d41e69e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9621f29237b4ee9ad63d107d41e69e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45799 } } }
I20260926 13:13:01.184966 20988 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:01.184973 20989 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:01.185243 20996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:01.186074 20996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:01.187901 20996 catalog_manager.cc:1347] Generated new cluster ID: a8c52409be7c407891bea55bb8ba8b78
I20260926 13:13:01.188011 20996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:01.188925 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:01.177941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45633 (local address 127.0.0.1:45799)
0926 13:13:01.178114 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:01.178122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:01.178148 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:13:01.178624 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:01.178632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:01.178682 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:01.178807 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:01.179640 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:01.180671 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:01.181806 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:01.182225 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:01.182309 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:01.185189 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:01.185194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:01.185195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:01.185207 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:01.187213 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:01.187719 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:01.187722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:01.187723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:01.187828 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:01.188079 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:01.188080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:01.188080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:01.188331 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:01.188478 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:01.188576 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 13:13:01.188711 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260926 13:13:01.189764 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:01.197168 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:01.191066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51319 (local address 127.0.0.1:45799)
0926 13:13:01.191103 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:01.191105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:01.191167 (+ 62us) server_negotiation.cc:407] Connection header received
0926 13:13:01.191234 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:01.191235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:01.191244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:01.191289 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:01.191815 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:01.192606 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:01.193339 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:01.193577 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:01.193633 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:01.194776 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:01.194778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:01.194778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:01.194785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:01.196152 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:01.196481 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:01.196482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:01.196483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:01.196538 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:01.196697 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:01.196697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:01.196698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:01.196813 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:01.196899 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:01.196992 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 13:13:01.197075 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:13:01.201037 20996 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:01.201586 20996 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:01.214423 20996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9621f29237b4ee9ad63d107d41e69e4: Generated new TSK 0
I20260926 13:13:01.214622 20996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:01.221164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 20915
2026-09-26T13:13:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1229 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:01 dist-test-slave-2dw5 krb5kdc[21014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:01 dist-test-slave-2dw5 krb5kdc[21014](info): set up 2 sockets
Sep 26 13:13:01 dist-test-slave-2dw5 krb5kdc[21014](info): commencing operation
krb5kdc: starting...
W20260926 13:13:02.182752 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:02 dist-test-slave-2dw5 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:13:02.300329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:02.387931 21027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:02.388145 21027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:02.388175 21027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:02.389357 21027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:02.389405 21027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:02.389422 21027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:02.389439 21027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:02.389454 21027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:02.391043 21027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:02.391289 21027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:02.391492 21027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:13:02.396492 21027 server_base.cc:1034] running on GCE node
W20260926 13:13:02.396471 21032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:02.396477 21033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:02.396674 21035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:02.396992 21027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:02.397339 21027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:02.398517 21027 hybrid_clock.cc:648] HybridClock initialized: now 1790428382398498 us; error 31 us; skew 500 ppm
Sep 26 13:13:02 dist-test-slave-2dw5 krb5kdc[21014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:02.401348 21027 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:13:02.401813 21027 webserver.cc:460] Webserver started at http://127.0.0.1:37443/ using document root <none> and password file <none>
I20260926 13:13:02.402009 21027 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:02.402055 21027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:02.402190 21027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:02.403335 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "c2910af97d4846acac0498865946d0c7"
format_stamp: "Formatted at 2026-09-26 13:13:02 on dist-test-slave-2dw5"
server_key: "e4b061863ad6171f54bf09e00f3ee598"
server_key_iv: "e38c73fce26e5300a24a3fc8051c8e79"
server_key_version: "encryptionkey@0"
I20260926 13:13:02.403662 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "c2910af97d4846acac0498865946d0c7"
format_stamp: "Formatted at 2026-09-26 13:13:02 on dist-test-slave-2dw5"
server_key: "e4b061863ad6171f54bf09e00f3ee598"
server_key_iv: "e38c73fce26e5300a24a3fc8051c8e79"
server_key_version: "encryptionkey@0"
I20260926 13:13:02.405090 21027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:13:02.405841 21042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:02.406006 21027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:13:02.406174 21027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "c2910af97d4846acac0498865946d0c7"
format_stamp: "Formatted at 2026-09-26 13:13:02 on dist-test-slave-2dw5"
server_key: "e4b061863ad6171f54bf09e00f3ee598"
server_key_iv: "e38c73fce26e5300a24a3fc8051c8e79"
server_key_version: "encryptionkey@0"
I20260926 13:13:02.406246 21027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:02.420783 21027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:02.422993 21027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:02.423142 21027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:02.426823 21027 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43707
I20260926 13:13:02.426879 21094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43707 every 8 connection(s)
I20260926 13:13:02.427243 21027 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:02.428123 21095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:02.430333 21095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7: Bootstrap starting.
I20260926 13:13:02.430904 21095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:02.431120 21095 log.cc:824] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:02.431779 21095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7: No bootstrap required, opened a new log
I20260926 13:13:02.433332 21095 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2910af97d4846acac0498865946d0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43707 } }
I20260926 13:13:02.433475 21095 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:02.433495 21095 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2910af97d4846acac0498865946d0c7, State: Initialized, Role: FOLLOWER
I20260926 13:13:02.433606 21095 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2910af97d4846acac0498865946d0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43707 } }
I20260926 13:13:02.433681 21095 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:02.433720 21095 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:02.433753 21095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:02.434227 21095 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2910af97d4846acac0498865946d0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43707 } }
I20260926 13:13:02.434312 21095 leader_election.cc:302] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2910af97d4846acac0498865946d0c7; no voters:
I20260926 13:13:02.434448 21095 leader_election.cc:288] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:02.434504 21098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:02.434696 21098 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [term 1 LEADER]: Becoming Leader. State: Replica: c2910af97d4846acac0498865946d0c7, State: Running, Role: LEADER
I20260926 13:13:02.434736 21095 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:02.434871 21098 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2910af97d4846acac0498865946d0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43707 } }
I20260926 13:13:02.435243 21099 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2910af97d4846acac0498865946d0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2910af97d4846acac0498865946d0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43707 } } }
I20260926 13:13:02.435292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 21027
I20260926 13:13:02.435317 21099 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:02.435406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:02.435534 21100 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2910af97d4846acac0498865946d0c7. Latest consensus state: current_term: 1 leader_uuid: "c2910af97d4846acac0498865946d0c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2910af97d4846acac0498865946d0c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43707 } } }
I20260926 13:13:02.435590 392 external_mini_cluster.cc:1428] Setting key ce9a4bac10fc3d357e9523ca2514cfb2
I20260926 13:13:02.435647 21100 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:02.435820 21102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:02.436420 21102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:02.438105 21102 catalog_manager.cc:1347] Generated new cluster ID: 147639ab27c046238126635a1225d4fb
I20260926 13:13:02.438164 21102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:02.440928 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:02.436243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52497 (local address 127.0.0.1:43707)
0926 13:13:02.436394 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:02.436404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:02.436625 (+ 221us) server_negotiation.cc:407] Connection header received
0926 13:13:02.436847 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:02.436855 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:02.436913 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:02.437064 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:02.437264 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:02.438315 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:02.439134 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:02.439589 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:02.439685 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:02.440339 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:02.440343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:02.440344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:02.440514 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:02.440545 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:02.440556 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:02.440610 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:02.440684 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 13:13:02.440758 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260926 13:13:02.441835 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:02.449796 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:02.443145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38607 (local address 127.0.0.1:43707)
0926 13:13:02.443180 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:02.443182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:02.447111 (+ 3929us) server_negotiation.cc:407] Connection header received
0926 13:13:02.447198 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:02.447199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:02.447220 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:02.447269 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:02.447404 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:02.447916 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:02.448641 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:02.448867 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:02.448925 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:02.449547 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:02.449549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:02.449549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:02.449554 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:02.449570 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:02.449574 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:02.449618 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:02.449676 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 13:13:02.449709 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:13:02.456786 21102 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:02.457587 21102 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:02.465044 21102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2910af97d4846acac0498865946d0c7: Generated new TSK 0
I20260926 13:13:02.465278 21102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:02.467481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 21027
2026-09-26T13:13:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1246 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:02 dist-test-slave-2dw5 krb5kdc[21126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:02 dist-test-slave-2dw5 krb5kdc[21126](info): set up 2 sockets
Sep 26 13:13:02 dist-test-slave-2dw5 krb5kdc[21126](info): commencing operation
krb5kdc: starting...
W20260926 13:13:03.426874 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:03 dist-test-slave-2dw5 krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:13:03.550096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:03.630730 21139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:03.630982 21139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:03.631016 21139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:03.632238 21139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:03.632289 21139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:03.632308 21139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:03.632323 21139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:03.632337 21139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:03.633896 21139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:03.634208 21139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:03.634485 21139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:03.639453 21144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:03.639453 21145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:03.639448 21147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:03.639845 21139 server_base.cc:1034] running on GCE node
I20260926 13:13:03.640072 21139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:03.640352 21139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:03.641515 21139 hybrid_clock.cc:648] HybridClock initialized: now 1790428383641499 us; error 32 us; skew 500 ppm
Sep 26 13:13:03 dist-test-slave-2dw5 krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:03.644068 21139 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:13:03.644546 21139 webserver.cc:460] Webserver started at http://127.0.0.1:39363/ using document root <none> and password file <none>
I20260926 13:13:03.644768 21139 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:03.644817 21139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:03.644942 21139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:03.646060 21139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "c4f1e61f3b8f4d12a3f8ac097644a79d"
format_stamp: "Formatted at 2026-09-26 13:13:03 on dist-test-slave-2dw5"
server_key: "3e7d8b4a9d7039f73f03455039dc3f84"
server_key_iv: "12edd1f7979d5dd61bba59e55fb23db8"
server_key_version: "encryptionkey@0"
I20260926 13:13:03.646386 21139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4f1e61f3b8f4d12a3f8ac097644a79d"
format_stamp: "Formatted at 2026-09-26 13:13:03 on dist-test-slave-2dw5"
server_key: "3e7d8b4a9d7039f73f03455039dc3f84"
server_key_iv: "12edd1f7979d5dd61bba59e55fb23db8"
server_key_version: "encryptionkey@0"
I20260926 13:13:03.647775 21139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:13:03.648665 21154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:03.648861 21139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:13:03.648952 21139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "c4f1e61f3b8f4d12a3f8ac097644a79d"
format_stamp: "Formatted at 2026-09-26 13:13:03 on dist-test-slave-2dw5"
server_key: "3e7d8b4a9d7039f73f03455039dc3f84"
server_key_iv: "12edd1f7979d5dd61bba59e55fb23db8"
server_key_version: "encryptionkey@0"
I20260926 13:13:03.649016 21139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:03.652563 21139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:03.654691 21139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:03.654829 21139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:03.658659 21139 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36285
I20260926 13:13:03.658696 21206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36285 every 8 connection(s)
I20260926 13:13:03.659060 21139 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:03.660075 21207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:03.662329 21207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d: Bootstrap starting.
I20260926 13:13:03.662935 21207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:03.663189 21207 log.cc:824] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:03.663880 21207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d: No bootstrap required, opened a new log
I20260926 13:13:03.664752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 21139
I20260926 13:13:03.664867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:03.665096 392 external_mini_cluster.cc:1428] Setting key 1457a160b75a13dd15296f7a13f615ae
I20260926 13:13:03.666007 21207 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f1e61f3b8f4d12a3f8ac097644a79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36285 } }
I20260926 13:13:03.666177 21207 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:03.666214 21207 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f1e61f3b8f4d12a3f8ac097644a79d, State: Initialized, Role: FOLLOWER
I20260926 13:13:03.666415 21207 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f1e61f3b8f4d12a3f8ac097644a79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36285 } }
I20260926 13:13:03.666510 21207 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:03.666548 21207 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:03.666599 21207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:03.667301 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:03.665663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46217 (local address 127.0.0.1:36285)
0926 13:13:03.665828 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:03.665836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:03.665861 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:13:03.665990 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:03.665995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:03.666020 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:03.666132 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:03.666796 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:03.666800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:03.666800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:03.666805 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:03.666878 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:03.666894 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:03.666968 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:03.667023 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 13:13:03.667078 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260926 13:13:03.667739 21207 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f1e61f3b8f4d12a3f8ac097644a79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36285 } }
I20260926 13:13:03.667846 21207 leader_election.cc:302] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4f1e61f3b8f4d12a3f8ac097644a79d; no voters:
I20260926 13:13:03.668000 21207 leader_election.cc:288] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:03.668021 21212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:03.668171 21212 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [term 1 LEADER]: Becoming Leader. State: Replica: c4f1e61f3b8f4d12a3f8ac097644a79d, State: Running, Role: LEADER
I20260926 13:13:03.668200 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:03.668246 21207 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:03.668273 21212 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f1e61f3b8f4d12a3f8ac097644a79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36285 } }
I20260926 13:13:03.668589 21212 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4f1e61f3b8f4d12a3f8ac097644a79d. Latest consensus state: current_term: 1 leader_uuid: "c4f1e61f3b8f4d12a3f8ac097644a79d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f1e61f3b8f4d12a3f8ac097644a79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36285 } } }
I20260926 13:13:03.668651 21212 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:03.668643 21213 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4f1e61f3b8f4d12a3f8ac097644a79d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f1e61f3b8f4d12a3f8ac097644a79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36285 } } }
I20260926 13:13:03.668740 21213 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:03.668864 21218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:03.669363 21218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:03.670754 21218 catalog_manager.cc:1347] Generated new cluster ID: 54a2d970caf0419ca220cb4906d68495
I20260926 13:13:03.670807 21218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:03.675271 21218 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:03.675504 21218 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:03.676079 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:03.670079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57127 (local address 127.0.0.1:36285)
0926 13:13:03.670111 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:03.670113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:03.675034 (+ 4921us) server_negotiation.cc:407] Connection header received
0926 13:13:03.675093 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:03.675094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:03.675100 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:03.675146 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:03.675864 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:03.675866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:03.675866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:03.675871 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:03.675884 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:03.675888 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:03.675915 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:03.676003 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 13:13:03.676015 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 13:13:03.681499 21218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4f1e61f3b8f4d12a3f8ac097644a79d: Generated new TSK 0
I20260926 13:13:03.681746 21218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:03.684369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 21139
2026-09-26T13:13:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:03 dist-test-slave-2dw5 krb5kdc[21238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:03 dist-test-slave-2dw5 krb5kdc[21238](info): set up 2 sockets
Sep 26 13:13:03 dist-test-slave-2dw5 krb5kdc[21238](info): commencing operation
krb5kdc: starting...
W20260926 13:13:04.650815 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:04 dist-test-slave-2dw5 krb5kdc[21238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:13:04.770313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46649
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:04.851027 21251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:04.851248 21251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:04.851271 21251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:04.852499 21251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:04.852550 21251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:04.852563 21251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:04.852581 21251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:04.852592 21251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:04.854125 21251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:04.854328 21251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:04.854518 21251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:04.859622 21256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:04.859640 21251 server_base.cc:1034] running on GCE node
W20260926 13:13:04.859622 21257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:04.859627 21259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:04.860119 21251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:04.860416 21251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:04.861565 21251 hybrid_clock.cc:648] HybridClock initialized: now 1790428384861561 us; error 40 us; skew 500 ppm
Sep 26 13:13:04 dist-test-slave-2dw5 krb5kdc[21238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:04.864032 21251 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:13:04.864445 21251 webserver.cc:460] Webserver started at http://127.0.0.1:33429/ using document root <none> and password file <none>
I20260926 13:13:04.864668 21251 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:04.864715 21251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:04.864809 21251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:04.865839 21251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "ab251d554d8e4c3b9391d7437ddf27da"
format_stamp: "Formatted at 2026-09-26 13:13:04 on dist-test-slave-2dw5"
server_key: "431879c484d1c09775de014a79e824ad"
server_key_iv: "b427f53bfc19d81c2d2c20002cc304f0"
server_key_version: "encryptionkey@0"
I20260926 13:13:04.866151 21251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab251d554d8e4c3b9391d7437ddf27da"
format_stamp: "Formatted at 2026-09-26 13:13:04 on dist-test-slave-2dw5"
server_key: "431879c484d1c09775de014a79e824ad"
server_key_iv: "b427f53bfc19d81c2d2c20002cc304f0"
server_key_version: "encryptionkey@0"
I20260926 13:13:04.867475 21251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:13:04.868197 21266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:04.868382 21251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:13:04.868474 21251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "ab251d554d8e4c3b9391d7437ddf27da"
format_stamp: "Formatted at 2026-09-26 13:13:04 on dist-test-slave-2dw5"
server_key: "431879c484d1c09775de014a79e824ad"
server_key_iv: "b427f53bfc19d81c2d2c20002cc304f0"
server_key_version: "encryptionkey@0"
I20260926 13:13:04.868543 21251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:04.892246 21251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:04.894325 21251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:04.894459 21251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:04.898437 21251 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46649
I20260926 13:13:04.898471 21318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46649 every 8 connection(s)
I20260926 13:13:04.898800 21251 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:04.899690 21319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:04.902029 21319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da: Bootstrap starting.
I20260926 13:13:04.902611 21319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:04.902911 21319 log.cc:824] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:04.903589 21319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da: No bootstrap required, opened a new log
I20260926 13:13:04.905233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 21251
I20260926 13:13:04.905326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:04.905249 21319 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab251d554d8e4c3b9391d7437ddf27da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46649 } }
I20260926 13:13:04.905421 21319 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:04.905468 21319 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab251d554d8e4c3b9391d7437ddf27da, State: Initialized, Role: FOLLOWER
I20260926 13:13:04.905499 392 external_mini_cluster.cc:1428] Setting key 693253eeaefbeabd5ff42b6053c20e87
I20260926 13:13:04.905608 21319 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab251d554d8e4c3b9391d7437ddf27da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46649 } }
I20260926 13:13:04.905682 21319 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:04.905720 21319 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:04.905766 21319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:04.906873 21319 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab251d554d8e4c3b9391d7437ddf27da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46649 } }
I20260926 13:13:04.906996 21319 leader_election.cc:302] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab251d554d8e4c3b9391d7437ddf27da; no voters:
I20260926 13:13:04.907145 21319 leader_election.cc:288] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:04.907182 21324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:04.907317 21324 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [term 1 LEADER]: Becoming Leader. State: Replica: ab251d554d8e4c3b9391d7437ddf27da, State: Running, Role: LEADER
I20260926 13:13:04.907388 21319 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:04.907462 21324 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab251d554d8e4c3b9391d7437ddf27da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46649 } }
I20260926 13:13:04.907866 21324 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab251d554d8e4c3b9391d7437ddf27da. Latest consensus state: current_term: 1 leader_uuid: "ab251d554d8e4c3b9391d7437ddf27da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab251d554d8e4c3b9391d7437ddf27da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46649 } } }
I20260926 13:13:04.907913 21326 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab251d554d8e4c3b9391d7437ddf27da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab251d554d8e4c3b9391d7437ddf27da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46649 } } }
I20260926 13:13:04.908010 21326 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:04.908190 21324 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:04.908329 21333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:04.909000 21333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:04.910671 21333 catalog_manager.cc:1347] Generated new cluster ID: 2f70d0412358404eb3ba66418feaab43
I20260926 13:13:04.910737 21333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:13:04 dist-test-slave-2dw5 krb5kdc[21238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:13:04.917802 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:04.905978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:39321 (local address 127.0.0.1:46649)
0926 13:13:04.906178 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:04.906188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:04.906224 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:13:04.906356 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:04.906364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:04.906430 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:04.906600 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:04.907273 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:04.908701 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:04.909885 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:04.910402 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:04.910509 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:04.913292 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:04.913297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:04.913297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:04.913310 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:04.915421 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:04.915969 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:04.915973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:04.915974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:04.916124 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:04.916374 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:04.916375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:04.916375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:04.916720 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:04.917321 (+ 601us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:04.917411 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 13:13:04.917535 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260926 13:13:04.918946 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:04.930339 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:04.920393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:45817 (local address 127.0.0.1:46649)
0926 13:13:04.920435 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:04.920436 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:04.923046 (+ 2610us) server_negotiation.cc:407] Connection header received
0926 13:13:04.923108 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:04.923109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:04.923122 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:04.923189 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:04.923872 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:04.924674 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:04.925719 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:04.926069 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:04.926140 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:04.927595 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:04.927597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:04.927597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:04.927604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:04.929125 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:04.929553 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:04.929555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:04.929555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:04.929619 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:04.929808 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:04.929809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:04.929810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:04.929947 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:04.930064 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:04.930168 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 13:13:04.930236 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 13:13:04.967608 21333 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:04.968128 21333 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:04.971946 21333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab251d554d8e4c3b9391d7437ddf27da: Generated new TSK 0
I20260926 13:13:04.972157 21333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:04.979832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 21251
W20260926 13:13:04.984043 21236 connection.cc:535] client connection to 127.0.0.1:46649 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46649): Connection reset by peer (error 104)
2026-09-26T13:13:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:05 dist-test-slave-2dw5 krb5kdc[21350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:05 dist-test-slave-2dw5 krb5kdc[21350](info): set up 2 sockets
Sep 26 13:13:05 dist-test-slave-2dw5 krb5kdc[21350](info): commencing operation
krb5kdc: starting...
W20260926 13:13:05.934767 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:06 dist-test-slave-2dw5 krb5kdc[21350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:13:06.053580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:06.134143 21363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:06.134383 21363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:06.134418 21363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:06.135676 21363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:06.135725 21363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:06.135743 21363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:06.135758 21363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:06.135774 21363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:06.137338 21363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:06.137598 21363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:06.137837 21363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:06.142645 21369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:06.142663 21371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:06.142663 21368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:06.142822 21363 server_base.cc:1034] running on GCE node
I20260926 13:13:06.143139 21363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:06.143415 21363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:06.144552 21363 hybrid_clock.cc:648] HybridClock initialized: now 1790428386144541 us; error 29 us; skew 500 ppm
Sep 26 13:13:06 dist-test-slave-2dw5 krb5kdc[21350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:06.147001 21363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:13:06.147462 21363 webserver.cc:460] Webserver started at http://127.0.0.1:38207/ using document root <none> and password file <none>
I20260926 13:13:06.147678 21363 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:06.147733 21363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:06.147862 21363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:06.149600 21363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "1e180a29e30c4ed6969222efe535d911"
format_stamp: "Formatted at 2026-09-26 13:13:06 on dist-test-slave-2dw5"
server_key: "2f33b19875a35ee95dad543d706ee973"
server_key_iv: "745f437529c50b4c81f28f1ec4c94b23"
server_key_version: "encryptionkey@0"
I20260926 13:13:06.149963 21363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e180a29e30c4ed6969222efe535d911"
format_stamp: "Formatted at 2026-09-26 13:13:06 on dist-test-slave-2dw5"
server_key: "2f33b19875a35ee95dad543d706ee973"
server_key_iv: "745f437529c50b4c81f28f1ec4c94b23"
server_key_version: "encryptionkey@0"
I20260926 13:13:06.151420 21363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260926 13:13:06.152184 21378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:06.152451 21363 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:13:06.152544 21363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "1e180a29e30c4ed6969222efe535d911"
format_stamp: "Formatted at 2026-09-26 13:13:06 on dist-test-slave-2dw5"
server_key: "2f33b19875a35ee95dad543d706ee973"
server_key_iv: "745f437529c50b4c81f28f1ec4c94b23"
server_key_version: "encryptionkey@0"
I20260926 13:13:06.152606 21363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:06.161233 21363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:06.163154 21363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:06.163287 21363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:06.167008 21363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42185
I20260926 13:13:06.167042 21430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42185 every 8 connection(s)
I20260926 13:13:06.167377 21363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:06.168212 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:06.168336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 21363
I20260926 13:13:06.168453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:06.168627 392 external_mini_cluster.cc:1428] Setting key 05199bb25f8974c377877e175a44c359
I20260926 13:13:06.171358 21431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911: Bootstrap starting.
I20260926 13:13:06.172101 21431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:06.172382 21431 log.cc:824] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:06.173012 21431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911: No bootstrap required, opened a new log
I20260926 13:13:06.173980 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:06.169135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:51921 (local address 127.0.0.1:42185)
0926 13:13:06.169299 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:06.169308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:06.169332 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:13:06.169771 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:06.169777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:06.169825 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:06.169946 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:06.170174 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:06.171118 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:06.172155 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:06.172611 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:06.172706 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:06.173391 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:06.173395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:06.173396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:06.173557 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:06.173586 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:06.173596 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:06.173665 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:06.173754 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 13:13:06.173781 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260926 13:13:06.175133 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:06.175117 21431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e180a29e30c4ed6969222efe535d911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42185 } }
I20260926 13:13:06.175287 21431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:06.175323 21431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e180a29e30c4ed6969222efe535d911, State: Initialized, Role: FOLLOWER
I20260926 13:13:06.175452 21431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e180a29e30c4ed6969222efe535d911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42185 } }
I20260926 13:13:06.175522 21431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:06.175554 21431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:06.175599 21431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:06.176306 21431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e180a29e30c4ed6969222efe535d911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42185 } }
I20260926 13:13:06.176415 21431 leader_election.cc:302] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e180a29e30c4ed6969222efe535d911; no voters:
I20260926 13:13:06.176576 21431 leader_election.cc:288] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:06.176678 21440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:06.176819 21440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [term 1 LEADER]: Becoming Leader. State: Replica: 1e180a29e30c4ed6969222efe535d911, State: Running, Role: LEADER
I20260926 13:13:06.176828 21431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:06.176967 21440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e180a29e30c4ed6969222efe535d911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42185 } }
I20260926 13:13:06.177270 21440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e180a29e30c4ed6969222efe535d911. Latest consensus state: current_term: 1 leader_uuid: "1e180a29e30c4ed6969222efe535d911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e180a29e30c4ed6969222efe535d911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42185 } } }
I20260926 13:13:06.177335 21440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:06.178166 21442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e180a29e30c4ed6969222efe535d911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e180a29e30c4ed6969222efe535d911" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42185 } } }
I20260926 13:13:06.178987 21442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911 [sys.catalog]: This master's current role is: LEADER
W20260926 13:13:06.178617 21457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:13:06.179709 21457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:13:06.179834 21447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:06.180066 21447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:06.180181 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:06.176611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:40349 (local address 127.0.0.1:42185)
0926 13:13:06.176663 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:06.176665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:06.176786 (+ 121us) server_negotiation.cc:407] Connection header received
0926 13:13:06.176832 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:06.176833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:06.176842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:06.176904 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:06.177153 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:06.177870 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:06.178823 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:06.179129 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:06.179197 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:06.179937 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:06.179939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:06.179939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:06.179944 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:06.179959 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:06.179964 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:06.180018 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:06.180100 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 13:13:06.180111 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:13:06.181656 21447 catalog_manager.cc:1347] Generated new cluster ID: 21fdd21945f74f28aed661583117218e
I20260926 13:13:06.181706 21447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:06.189944 21447 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:06.190711 21447 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:06.197592 21447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e180a29e30c4ed6969222efe535d911: Generated new TSK 0
I20260926 13:13:06.197810 21447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:06.200802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 21363
W20260926 13:13:06.204571 21348 connection.cc:535] client connection to 127.0.0.1:42185 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42185): Connection reset by peer (error 104)
2026-09-26T13:13:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:06 dist-test-slave-2dw5 krb5kdc[21462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:06 dist-test-slave-2dw5 krb5kdc[21462](info): set up 2 sockets
Sep 26 13:13:06 dist-test-slave-2dw5 krb5kdc[21462](info): commencing operation
krb5kdc: starting...
W20260926 13:13:07.155007 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:07 dist-test-slave-2dw5 krb5kdc[21462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:13:07.273180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:07.357503 21475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:07.357752 21475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:07.357787 21475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:07.359113 21475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:07.359174 21475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:07.359193 21475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:07.359210 21475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:07.359225 21475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:07.360839 21475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:07.361088 21475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:07.361300 21475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:07.366304 21483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:07.366314 21481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:07.366322 21475 server_base.cc:1034] running on GCE node
W20260926 13:13:07.366304 21480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:07.366873 21475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:07.367149 21475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:07.368281 21475 hybrid_clock.cc:648] HybridClock initialized: now 1790428387368270 us; error 31 us; skew 500 ppm
Sep 26 13:13:07 dist-test-slave-2dw5 krb5kdc[21462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:07.370714 21475 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:13:07.371206 21475 webserver.cc:460] Webserver started at http://127.0.0.1:37387/ using document root <none> and password file <none>
I20260926 13:13:07.371392 21475 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:07.371431 21475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:07.371520 21475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:07.372529 21475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "a12b305e83c74d01866f83705eef2be4"
format_stamp: "Formatted at 2026-09-26 13:13:07 on dist-test-slave-2dw5"
server_key: "ff5bad11b82d772b4701ca0f2896e3ef"
server_key_iv: "31172d9f6a1124d46eb3f987ea00b295"
server_key_version: "encryptionkey@0"
I20260926 13:13:07.372843 21475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "a12b305e83c74d01866f83705eef2be4"
format_stamp: "Formatted at 2026-09-26 13:13:07 on dist-test-slave-2dw5"
server_key: "ff5bad11b82d772b4701ca0f2896e3ef"
server_key_iv: "31172d9f6a1124d46eb3f987ea00b295"
server_key_version: "encryptionkey@0"
I20260926 13:13:07.374027 21475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:13:07.374675 21490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:07.374902 21475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:07.374982 21475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "a12b305e83c74d01866f83705eef2be4"
format_stamp: "Formatted at 2026-09-26 13:13:07 on dist-test-slave-2dw5"
server_key: "ff5bad11b82d772b4701ca0f2896e3ef"
server_key_iv: "31172d9f6a1124d46eb3f987ea00b295"
server_key_version: "encryptionkey@0"
I20260926 13:13:07.375049 21475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:07.379665 21475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:07.382007 21475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:07.382129 21475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:07.385990 21475 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38091
I20260926 13:13:07.386076 21542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38091 every 8 connection(s)
I20260926 13:13:07.386348 21475 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:07.387477 21543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:07.388010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 21475
I20260926 13:13:07.388135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:07.388295 392 external_mini_cluster.cc:1428] Setting key d571873b92075d016d2be02502bcc9c5
I20260926 13:13:07.390374 21543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4: Bootstrap starting.
I20260926 13:13:07.390923 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:07.388823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:50435 (local address 127.0.0.1:38091)
0926 13:13:07.389003 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:07.389013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:07.389045 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:13:07.389587 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:07.389595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:07.389633 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:07.389788 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:07.390472 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:07.390476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:07.390477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:07.390481 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:07.390524 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:07.390537 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:07.390609 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:07.390657 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 13:13:07.390703 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260926 13:13:07.391113 21543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:07.391410 21543 log.cc:824] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:07.391842 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:07.392076 21543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4: No bootstrap required, opened a new log
I20260926 13:13:07.394253 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:07.393131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:48831 (local address 127.0.0.1:38091)
0926 13:13:07.393165 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:07.393167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:07.393279 (+ 112us) server_negotiation.cc:407] Connection header received
0926 13:13:07.393345 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:07.393346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:07.393350 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:07.393396 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:07.394059 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:07.394061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:07.394061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:07.394065 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:07.394079 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:07.394083 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:07.394163 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:07.394178 (+ 15us) server_negotiation.cc:299] Negotiation successful
0926 13:13:07.394190 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 13:13:07.394156 21543 raft_consensus.cc:348] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12b305e83c74d01866f83705eef2be4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38091 } }
I20260926 13:13:07.394320 21543 raft_consensus.cc:374] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:07.394356 21543 raft_consensus.cc:729] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12b305e83c74d01866f83705eef2be4, State: Initialized, Role: FOLLOWER
I20260926 13:13:07.394469 21543 consensus_queue.cc:260] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12b305e83c74d01866f83705eef2be4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38091 } }
I20260926 13:13:07.394546 21543 raft_consensus.cc:388] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:07.394577 21543 raft_consensus.cc:482] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:07.394632 21543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:07.395277 21543 raft_consensus.cc:504] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12b305e83c74d01866f83705eef2be4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38091 } }
I20260926 13:13:07.395371 21543 leader_election.cc:302] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a12b305e83c74d01866f83705eef2be4; no voters:
I20260926 13:13:07.395532 21543 leader_election.cc:288] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:07.395617 21553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:07.395759 21553 raft_consensus.cc:686] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [term 1 LEADER]: Becoming Leader. State: Replica: a12b305e83c74d01866f83705eef2be4, State: Running, Role: LEADER
I20260926 13:13:07.395784 21543 sys_catalog.cc:564] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:07.395953 21553 consensus_queue.cc:237] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12b305e83c74d01866f83705eef2be4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38091 } }
I20260926 13:13:07.396325 21555 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a12b305e83c74d01866f83705eef2be4. Latest consensus state: current_term: 1 leader_uuid: "a12b305e83c74d01866f83705eef2be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12b305e83c74d01866f83705eef2be4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38091 } } }
I20260926 13:13:07.396356 21554 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a12b305e83c74d01866f83705eef2be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12b305e83c74d01866f83705eef2be4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38091 } } }
I20260926 13:13:07.396476 21554 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:07.396713 21555 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:07.396716 21561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:07.397435 21561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:07.399055 21561 catalog_manager.cc:1347] Generated new cluster ID: ef8b5b3c36de48ae81274dbafda21810
I20260926 13:13:07.399114 21561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:07.409307 21561 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:07.409518 21561 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:07.422137 21561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a12b305e83c74d01866f83705eef2be4: Generated new TSK 0
I20260926 13:13:07.422369 21561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:07.423866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 21475
2026-09-26T13:13:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:07 dist-test-slave-2dw5 krb5kdc[21574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:07 dist-test-slave-2dw5 krb5kdc[21574](info): set up 2 sockets
Sep 26 13:13:07 dist-test-slave-2dw5 krb5kdc[21574](info): commencing operation
krb5kdc: starting...
W20260926 13:13:08.378966 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:08 dist-test-slave-2dw5 krb5kdc[21574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:13:08.498786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:08.578893 21587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:08.579149 21587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:08.579173 21587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:08.580370 21587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:08.580410 21587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:08.580423 21587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:08.580435 21587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:08.580444 21587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:08.582098 21587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:08.582312 21587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:08.582494 21587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:08.587466 21593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:08.587466 21595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:08.587634 21587 server_base.cc:1034] running on GCE node
W20260926 13:13:08.587471 21592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:08.588132 21587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:08.588455 21587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:08.589612 21587 hybrid_clock.cc:648] HybridClock initialized: now 1790428388589593 us; error 34 us; skew 500 ppm
Sep 26 13:13:08 dist-test-slave-2dw5 krb5kdc[21574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:08.592320 21587 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:13:08.592753 21587 webserver.cc:460] Webserver started at http://127.0.0.1:39197/ using document root <none> and password file <none>
I20260926 13:13:08.592959 21587 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:08.593006 21587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:08.593151 21587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:08.594206 21587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "c908a8f4e31143c7981532f5886d60d3"
format_stamp: "Formatted at 2026-09-26 13:13:08 on dist-test-slave-2dw5"
server_key: "cc6681d41883bd3b485971b86ea96acc"
server_key_iv: "7effd364c50bcc2ef443fe607252c151"
server_key_version: "encryptionkey@0"
I20260926 13:13:08.594537 21587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "c908a8f4e31143c7981532f5886d60d3"
format_stamp: "Formatted at 2026-09-26 13:13:08 on dist-test-slave-2dw5"
server_key: "cc6681d41883bd3b485971b86ea96acc"
server_key_iv: "7effd364c50bcc2ef443fe607252c151"
server_key_version: "encryptionkey@0"
I20260926 13:13:08.596055 21587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:13:08.596884 21602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:08.597044 21587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:13:08.597211 21587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "c908a8f4e31143c7981532f5886d60d3"
format_stamp: "Formatted at 2026-09-26 13:13:08 on dist-test-slave-2dw5"
server_key: "cc6681d41883bd3b485971b86ea96acc"
server_key_iv: "7effd364c50bcc2ef443fe607252c151"
server_key_version: "encryptionkey@0"
I20260926 13:13:08.597285 21587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:08.612896 21587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:08.615307 21587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:08.615482 21587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:08.619738 21587 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41437
I20260926 13:13:08.619767 21654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41437 every 8 connection(s)
I20260926 13:13:08.620144 21587 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:08.621001 21655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:08.623597 21655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3: Bootstrap starting.
I20260926 13:13:08.623729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 21587
I20260926 13:13:08.623858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:08.624061 392 external_mini_cluster.cc:1428] Setting key e64cabfe32a9971162735b92448340e6
I20260926 13:13:08.624219 21655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:08.624516 21655 log.cc:824] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:08.625252 21655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3: No bootstrap required, opened a new log
I20260926 13:13:08.627506 21655 raft_consensus.cc:348] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908a8f4e31143c7981532f5886d60d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } }
I20260926 13:13:08.627682 21655 raft_consensus.cc:374] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:08.627727 21655 raft_consensus.cc:729] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c908a8f4e31143c7981532f5886d60d3, State: Initialized, Role: FOLLOWER
I20260926 13:13:08.627863 21655 consensus_queue.cc:260] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908a8f4e31143c7981532f5886d60d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } }
I20260926 13:13:08.627924 21655 raft_consensus.cc:388] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:08.627974 21655 raft_consensus.cc:482] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:08.628027 21655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:08.628633 21655 raft_consensus.cc:504] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908a8f4e31143c7981532f5886d60d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } }
I20260926 13:13:08.628737 21655 leader_election.cc:302] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c908a8f4e31143c7981532f5886d60d3; no voters:
I20260926 13:13:08.628902 21655 leader_election.cc:288] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:08.628947 21660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:08.629091 21660 raft_consensus.cc:686] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [term 1 LEADER]: Becoming Leader. State: Replica: c908a8f4e31143c7981532f5886d60d3, State: Running, Role: LEADER
I20260926 13:13:08.629179 21655 sys_catalog.cc:564] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:08.629247 21660 consensus_queue.cc:237] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908a8f4e31143c7981532f5886d60d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } }
I20260926 13:13:08.630663 21661 sys_catalog.cc:455] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c908a8f4e31143c7981532f5886d60d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908a8f4e31143c7981532f5886d60d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } } }
I20260926 13:13:08.630692 21662 sys_catalog.cc:455] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c908a8f4e31143c7981532f5886d60d3. Latest consensus state: current_term: 1 leader_uuid: "c908a8f4e31143c7981532f5886d60d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c908a8f4e31143c7981532f5886d60d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41437 } } }
I20260926 13:13:08.630780 21661 sys_catalog.cc:458] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:08.630788 21662 sys_catalog.cc:458] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:08.631110 21669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 13:13:08 dist-test-slave-2dw5 krb5kdc[21574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:13:08.631558 21669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:08.633266 21669 catalog_manager.cc:1347] Generated new cluster ID: 5f1324baf29d451781e88a5561c4dac8
I20260926 13:13:08.633308 21669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:08.636682 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:08.624579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:51751 (local address 127.0.0.1:41437)
0926 13:13:08.624761 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:08.624770 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:08.624802 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:13:08.625362 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:08.625371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:08.625435 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:08.625592 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:08.626160 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:08.627567 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:08.628417 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:08.629027 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:08.629466 (+ 439us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:08.631984 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:08.631989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:08.631990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:08.632003 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:08.634897 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:08.635364 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:08.635367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:08.635368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:08.635504 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:08.635728 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:08.635730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:08.635730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:08.636066 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:08.636245 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:08.636331 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 13:13:08.636445 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260926 13:13:08.637687 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:08.654472 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:08.639398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:46211 (local address 127.0.0.1:41437)
0926 13:13:08.639440 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:08.639442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:08.647057 (+ 7615us) server_negotiation.cc:407] Connection header received
0926 13:13:08.647136 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:08.647137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:08.647152 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:08.647212 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:08.647879 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:08.648700 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:08.649718 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:08.650075 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:08.650148 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:08.651614 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:08.651615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:08.651616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:08.651623 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:08.653205 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:08.653691 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:08.653692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:08.653692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:08.653758 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:08.653950 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:08.653951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:08.653951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:08.654091 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:08.654194 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:08.654301 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 13:13:08.654370 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:13:08.661988 21669 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:08.662556 21669 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:08.671482 21669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c908a8f4e31143c7981532f5886d60d3: Generated new TSK 0
I20260926 13:13:08.671767 21669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:08.679543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 21587
W20260926 13:13:08.683706 21572 connection.cc:535] client connection to 127.0.0.1:41437 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41437): Connection reset by peer (error 104)
2026-09-26T13:13:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:08 dist-test-slave-2dw5 krb5kdc[21686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:08 dist-test-slave-2dw5 krb5kdc[21686](info): set up 2 sockets
Sep 26 13:13:08 dist-test-slave-2dw5 krb5kdc[21686](info): commencing operation
krb5kdc: starting...
W20260926 13:13:09.646898 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:09 dist-test-slave-2dw5 krb5kdc[21686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:13:09.766016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38303
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:09.846274 21699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:09.846494 21699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:09.846517 21699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:09.847738 21699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:09.847785 21699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:09.847800 21699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:09.847810 21699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:09.847821 21699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:09.849366 21699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38303
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:09.849609 21699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:09.849814 21699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:09.854619 21707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:09.854740 21704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:09.854626 21705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:09.854938 21699 server_base.cc:1034] running on GCE node
I20260926 13:13:09.855187 21699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:09.855475 21699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:09.856736 21699 hybrid_clock.cc:648] HybridClock initialized: now 1790428389856708 us; error 31 us; skew 500 ppm
Sep 26 13:13:09 dist-test-slave-2dw5 krb5kdc[21686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:09.859371 21699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:13:09.859860 21699 webserver.cc:460] Webserver started at http://127.0.0.1:46153/ using document root <none> and password file <none>
I20260926 13:13:09.860124 21699 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:09.860186 21699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:09.860317 21699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:09.861464 21699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "8ec86c8e84ba47bb9e338fb57e800266"
format_stamp: "Formatted at 2026-09-26 13:13:09 on dist-test-slave-2dw5"
server_key: "af34b34333b0390254fb6ed488a2cb9c"
server_key_iv: "7a1c7ddb5adbb7c0c9e1cd597e622447"
server_key_version: "encryptionkey@0"
I20260926 13:13:09.861789 21699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ec86c8e84ba47bb9e338fb57e800266"
format_stamp: "Formatted at 2026-09-26 13:13:09 on dist-test-slave-2dw5"
server_key: "af34b34333b0390254fb6ed488a2cb9c"
server_key_iv: "7a1c7ddb5adbb7c0c9e1cd597e622447"
server_key_version: "encryptionkey@0"
I20260926 13:13:09.863147 21699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:13:09.863884 21714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:09.864077 21699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:13:09.864161 21699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "8ec86c8e84ba47bb9e338fb57e800266"
format_stamp: "Formatted at 2026-09-26 13:13:09 on dist-test-slave-2dw5"
server_key: "af34b34333b0390254fb6ed488a2cb9c"
server_key_iv: "7a1c7ddb5adbb7c0c9e1cd597e622447"
server_key_version: "encryptionkey@0"
I20260926 13:13:09.864219 21699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:09.875989 21699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:09.878077 21699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:09.878187 21699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:09.881953 21699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38303
I20260926 13:13:09.882012 21766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38303 every 8 connection(s)
I20260926 13:13:09.882301 21699 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:09.883302 21767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:09.885597 21767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266: Bootstrap starting.
I20260926 13:13:09.886142 21767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:09.886407 21767 log.cc:824] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:09.887231 21767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266: No bootstrap required, opened a new log
I20260926 13:13:09.888974 21767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec86c8e84ba47bb9e338fb57e800266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38303 } }
I20260926 13:13:09.889124 21767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:09.889156 21767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec86c8e84ba47bb9e338fb57e800266, State: Initialized, Role: FOLLOWER
I20260926 13:13:09.889293 21767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec86c8e84ba47bb9e338fb57e800266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38303 } }
I20260926 13:13:09.889369 21767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:09.889400 21767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:09.889457 21767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:09.890036 21767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec86c8e84ba47bb9e338fb57e800266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38303 } }
I20260926 13:13:09.890139 21767 leader_election.cc:302] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ec86c8e84ba47bb9e338fb57e800266; no voters:
I20260926 13:13:09.890321 21767 leader_election.cc:288] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:09.890388 21770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:09.890560 21770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [term 1 LEADER]: Becoming Leader. State: Replica: 8ec86c8e84ba47bb9e338fb57e800266, State: Running, Role: LEADER
I20260926 13:13:09.890604 21767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:09.890669 21770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec86c8e84ba47bb9e338fb57e800266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38303 } }
I20260926 13:13:09.890738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 21699
I20260926 13:13:09.890911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:09.891168 392 external_mini_cluster.cc:1428] Setting key 851e9969199a13287ed144fea288e1b6
I20260926 13:13:09.891258 21771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ec86c8e84ba47bb9e338fb57e800266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec86c8e84ba47bb9e338fb57e800266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38303 } } }
I20260926 13:13:09.891328 21771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:09.891808 21772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ec86c8e84ba47bb9e338fb57e800266. Latest consensus state: current_term: 1 leader_uuid: "8ec86c8e84ba47bb9e338fb57e800266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec86c8e84ba47bb9e338fb57e800266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38303 } } }
I20260926 13:13:09.891891 21772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:09.892025 21779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:09.892604 21779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:09.894415 21779 catalog_manager.cc:1347] Generated new cluster ID: 5a786edb6f244d9399dc93ed5c3508d1
I20260926 13:13:09.894577 21779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:09.897253 21782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:09.891620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:58877 (local address 127.0.0.1:38303)
0926 13:13:09.891925 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:09.891948 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:09.891985 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:13:09.892124 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:09.892131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:09.892195 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:09.892336 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:09.892585 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:09.893808 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:09.894950 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:09.895684 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:09.895791 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:09.896510 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:09.896514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:09.896514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:09.896747 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:09.896791 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:09.896805 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:09.896872 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:09.897002 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 13:13:09.897030 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":82,"threads_started":1}
I20260926 13:13:09.898217 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:09.903301 21782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:09.899917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:34743 (local address 127.0.0.1:38303)
0926 13:13:09.899951 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:09.899953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:09.899961 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:13:09.900029 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:09.900030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:09.900039 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:09.900089 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:09.900215 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:09.900953 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:09.901959 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:09.902261 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:09.902317 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:09.903051 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:09.903052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:09.903053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:09.903058 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:09.903073 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:09.903077 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:09.903127 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:09.903193 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 13:13:09.903206 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 13:13:09.906505 21779 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:09.907282 21779 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:09.915702 21779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ec86c8e84ba47bb9e338fb57e800266: Generated new TSK 0
I20260926 13:13:09.915944 21779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:09.923772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 21699
W20260926 13:13:09.927561 21684 connection.cc:535] client connection to 127.0.0.1:38303 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38303): Connection reset by peer (error 104)
2026-09-26T13:13:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1243 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:09 dist-test-slave-2dw5 krb5kdc[21798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:09 dist-test-slave-2dw5 krb5kdc[21798](info): set up 2 sockets
Sep 26 13:13:09 dist-test-slave-2dw5 krb5kdc[21798](info): commencing operation
krb5kdc: starting...
W20260926 13:13:10.895025 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:10 dist-test-slave-2dw5 krb5kdc[21798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:13:11.014930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:11.096922 21811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:11.097157 21811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:11.097183 21811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:11.098397 21811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:11.098440 21811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:11.098453 21811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:11.098465 21811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:11.098476 21811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:11.100127 21811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:11.100353 21811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:11.100536 21811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:11.105486 21816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:11.105525 21817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:11.105557 21811 server_base.cc:1034] running on GCE node
W20260926 13:13:11.105526 21819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:11.105924 21811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:11.106225 21811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:11.107394 21811 hybrid_clock.cc:648] HybridClock initialized: now 1790428391107362 us; error 42 us; skew 500 ppm
Sep 26 13:13:11 dist-test-slave-2dw5 krb5kdc[21798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:11.109949 21811 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:13:11.110414 21811 webserver.cc:460] Webserver started at http://127.0.0.1:40565/ using document root <none> and password file <none>
I20260926 13:13:11.110646 21811 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:11.110745 21811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:11.110910 21811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:11.112020 21811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "f6ac77c247b445a39edceaaf8a1d4d38"
format_stamp: "Formatted at 2026-09-26 13:13:11 on dist-test-slave-2dw5"
server_key: "98a1423f95be29a1034ce959cacd8e87"
server_key_iv: "061fb017024f7d2f32c1a96ea32b94ca"
server_key_version: "encryptionkey@0"
I20260926 13:13:11.112355 21811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6ac77c247b445a39edceaaf8a1d4d38"
format_stamp: "Formatted at 2026-09-26 13:13:11 on dist-test-slave-2dw5"
server_key: "98a1423f95be29a1034ce959cacd8e87"
server_key_iv: "061fb017024f7d2f32c1a96ea32b94ca"
server_key_version: "encryptionkey@0"
I20260926 13:13:11.113595 21811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:13:11.114261 21826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:11.114475 21811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:13:11.114565 21811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "f6ac77c247b445a39edceaaf8a1d4d38"
format_stamp: "Formatted at 2026-09-26 13:13:11 on dist-test-slave-2dw5"
server_key: "98a1423f95be29a1034ce959cacd8e87"
server_key_iv: "061fb017024f7d2f32c1a96ea32b94ca"
server_key_version: "encryptionkey@0"
I20260926 13:13:11.114629 21811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:11.118070 21811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:11.120229 21811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:11.120368 21811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:11.124318 21811 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35827
I20260926 13:13:11.124368 21878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35827 every 8 connection(s)
I20260926 13:13:11.124691 21811 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:11.125808 21879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:11.128129 21879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38: Bootstrap starting.
I20260926 13:13:11.128664 21879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:11.128923 21879 log.cc:824] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:11.129608 21879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38: No bootstrap required, opened a new log
I20260926 13:13:11.129680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 21811
I20260926 13:13:11.129799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:11.129997 392 external_mini_cluster.cc:1428] Setting key b28b6815bf94038b2966c373e0e7a4ad
I20260926 13:13:11.131615 21879 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ac77c247b445a39edceaaf8a1d4d38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35827 } }
I20260926 13:13:11.131757 21879 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:11.131789 21879 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ac77c247b445a39edceaaf8a1d4d38, State: Initialized, Role: FOLLOWER
I20260926 13:13:11.131946 21879 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ac77c247b445a39edceaaf8a1d4d38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35827 } }
I20260926 13:13:11.132053 21879 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:11.132098 21879 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:11.132143 21879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:11.132743 21879 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ac77c247b445a39edceaaf8a1d4d38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35827 } }
I20260926 13:13:11.132841 21879 leader_election.cc:302] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6ac77c247b445a39edceaaf8a1d4d38; no voters:
I20260926 13:13:11.132995 21879 leader_election.cc:288] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:11.133033 21884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:11.133178 21884 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [term 1 LEADER]: Becoming Leader. State: Replica: f6ac77c247b445a39edceaaf8a1d4d38, State: Running, Role: LEADER
I20260926 13:13:11.133306 21879 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:11.133302 21884 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ac77c247b445a39edceaaf8a1d4d38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35827 } }
I20260926 13:13:11.133378 21883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:11.130494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:60377 (local address 127.0.0.1:35827)
0926 13:13:11.130674 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:11.130683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:11.130717 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:13:11.131620 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:11.131629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:11.131660 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:11.131823 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:11.132761 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:11.132764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:11.132764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:11.132768 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:11.132808 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:11.132916 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:11.132981 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:11.133046 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 13:13:11.133180 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260926 13:13:11.133709 21886 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6ac77c247b445a39edceaaf8a1d4d38. Latest consensus state: current_term: 1 leader_uuid: "f6ac77c247b445a39edceaaf8a1d4d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ac77c247b445a39edceaaf8a1d4d38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35827 } } }
I20260926 13:13:11.133761 21885 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6ac77c247b445a39edceaaf8a1d4d38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ac77c247b445a39edceaaf8a1d4d38" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35827 } } }
I20260926 13:13:11.134217 21886 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:11.134228 21885 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:11.134384 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:11.134524 21893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:11.135043 21893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:11.140010 21893 catalog_manager.cc:1347] Generated new cluster ID: 1fa8b6235f514ab4ac33b12eed5f4549
I20260926 13:13:11.140076 21893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:11.140376 21883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:11.135941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:50997 (local address 127.0.0.1:35827)
0926 13:13:11.135984 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:11.135987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:11.139464 (+ 3477us) server_negotiation.cc:407] Connection header received
0926 13:13:11.139536 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:11.139537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:11.139544 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:11.139590 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:11.140146 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:11.140147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:11.140148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:11.140152 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:11.140169 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:11.140174 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:11.140202 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:11.140289 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 13:13:11.140302 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:13:11.160248 21893 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:11.160449 21893 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:11.166385 21893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6ac77c247b445a39edceaaf8a1d4d38: Generated new TSK 0
I20260926 13:13:11.166606 21893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:11.175886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 21811
2026-09-26T13:13:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:11 dist-test-slave-2dw5 krb5kdc[21910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:11 dist-test-slave-2dw5 krb5kdc[21910](info): set up 2 sockets
Sep 26 13:13:11 dist-test-slave-2dw5 krb5kdc[21910](info): commencing operation
krb5kdc: starting...
W20260926 13:13:12.143167 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:12 dist-test-slave-2dw5 krb5kdc[21910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:13:12.262459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44711
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:12.345010 21923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:12.345232 21923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:12.345263 21923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:12.346484 21923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:12.346532 21923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:12.346551 21923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:12.346566 21923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:12.346583 21923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:12.348162 21923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:12.348430 21923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:12.348637 21923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:12.353384 21931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:12.353495 21929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:12.353528 21923 server_base.cc:1034] running on GCE node
W20260926 13:13:12.353390 21928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:12.353837 21923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:12.354107 21923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:12.355311 21923 hybrid_clock.cc:648] HybridClock initialized: now 1790428392355293 us; error 37 us; skew 500 ppm
Sep 26 13:13:12 dist-test-slave-2dw5 krb5kdc[21910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:12.357827 21923 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:13:12.358268 21923 webserver.cc:460] Webserver started at http://127.0.98.62:37065/ using document root <none> and password file <none>
I20260926 13:13:12.358474 21923 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:12.358515 21923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:12.358601 21923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:12.359670 21923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "3de7e53e48e3451994e63fc0ebd288ce"
format_stamp: "Formatted at 2026-09-26 13:13:12 on dist-test-slave-2dw5"
server_key: "0ffdc35e544279176100686e85e55d2e"
server_key_iv: "c2bc3e15470384f9570cb214ee1a4b23"
server_key_version: "encryptionkey@0"
I20260926 13:13:12.359972 21923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "3de7e53e48e3451994e63fc0ebd288ce"
format_stamp: "Formatted at 2026-09-26 13:13:12 on dist-test-slave-2dw5"
server_key: "0ffdc35e544279176100686e85e55d2e"
server_key_iv: "c2bc3e15470384f9570cb214ee1a4b23"
server_key_version: "encryptionkey@0"
I20260926 13:13:12.361229 21923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:13:12.361955 21938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:12.362134 21923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:13:12.362219 21923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "3de7e53e48e3451994e63fc0ebd288ce"
format_stamp: "Formatted at 2026-09-26 13:13:12 on dist-test-slave-2dw5"
server_key: "0ffdc35e544279176100686e85e55d2e"
server_key_iv: "c2bc3e15470384f9570cb214ee1a4b23"
server_key_version: "encryptionkey@0"
I20260926 13:13:12.362284 21923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:12.372882 21923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:12.375020 21923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:12.375156 21923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:12.379317 21923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44711
I20260926 13:13:12.379366 21990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44711 every 8 connection(s)
I20260926 13:13:12.379694 21923 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:12.380581 21991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:12.382799 21991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce: Bootstrap starting.
I20260926 13:13:12.383389 21991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:12.383636 21991 log.cc:824] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:12.384400 21991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce: No bootstrap required, opened a new log
I20260926 13:13:12.386072 21991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de7e53e48e3451994e63fc0ebd288ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260926 13:13:12.386222 21991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:12.386253 21991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3de7e53e48e3451994e63fc0ebd288ce, State: Initialized, Role: FOLLOWER
I20260926 13:13:12.386395 21991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de7e53e48e3451994e63fc0ebd288ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260926 13:13:12.386463 21991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:12.386487 21991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:12.386531 21991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:12.387212 21991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de7e53e48e3451994e63fc0ebd288ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260926 13:13:12.387315 21991 leader_election.cc:302] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3de7e53e48e3451994e63fc0ebd288ce; no voters:
I20260926 13:13:12.387494 21991 leader_election.cc:288] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:12.387563 21994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:12.387722 21994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [term 1 LEADER]: Becoming Leader. State: Replica: 3de7e53e48e3451994e63fc0ebd288ce, State: Running, Role: LEADER
I20260926 13:13:12.387727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 21923
I20260926 13:13:12.387802 21991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:12.387862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:12.387876 21994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de7e53e48e3451994e63fc0ebd288ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } }
I20260926 13:13:12.388046 392 external_mini_cluster.cc:1428] Setting key 25d7e9747e68533d4b2a4244afcf7704
I20260926 13:13:12.388293 21995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3de7e53e48e3451994e63fc0ebd288ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de7e53e48e3451994e63fc0ebd288ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } } }
I20260926 13:13:12.388370 21995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:12.388406 21996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3de7e53e48e3451994e63fc0ebd288ce. Latest consensus state: current_term: 1 leader_uuid: "3de7e53e48e3451994e63fc0ebd288ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3de7e53e48e3451994e63fc0ebd288ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44711 } } }
I20260926 13:13:12.388484 21996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:12.388681 22002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:12.389230 22002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:12.391263 22002 catalog_manager.cc:1347] Generated new cluster ID: 69bdfd2c117442dcb6f94ec2683d2e4d
I20260926 13:13:12.391316 22002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:13:12 dist-test-slave-2dw5 krb5kdc[21910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:13:12.397485 22002 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:12.398173 22002 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:12.400439 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:12.388671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56769 (local address 127.0.98.62:44711)
0926 13:13:12.389062 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:12.389071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:12.389100 (+ 29us) server_negotiation.cc:407] Connection header received
0926 13:13:12.389225 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:12.389232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:12.389356 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:12.389500 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:12.390229 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:12.391581 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:12.392745 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:12.393461 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:12.393584 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:12.396513 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:12.396518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:12.396518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:12.396533 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:12.398639 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:12.399201 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:12.399204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:12.399205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:12.399346 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:12.399582 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:12.399583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:12.399583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:12.399847 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:12.399988 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:12.400111 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 13:13:12.400211 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":87,"threads_started":1}
I20260926 13:13:12.401458 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:12.402686 22002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3de7e53e48e3451994e63fc0ebd288ce: Generated new TSK 0
I20260926 13:13:12.402872 22002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:12.409675 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:12.402975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54943 (local address 127.0.98.62:44711)
0926 13:13:12.403021 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:12.403022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:12.403128 (+ 106us) server_negotiation.cc:407] Connection header received
0926 13:13:12.403194 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:12.403194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:12.403202 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:12.403247 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:12.403843 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:12.404450 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:12.405547 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:12.405754 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:12.405806 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:12.407226 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:12.407228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:12.407228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:12.407234 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:12.408514 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:12.408926 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:12.408927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:12.408927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:12.408973 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:12.409214 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:12.409215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:12.409215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:12.409335 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:12.409427 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:12.409558 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 13:13:12.409609 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 13:13:12.411787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 21923
W20260926 13:13:12.415596 21908 connection.cc:535] client connection to 127.0.98.62:44711 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44711): Connection reset by peer (error 104)
2026-09-26T13:13:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:12 dist-test-slave-2dw5 krb5kdc[22022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:12 dist-test-slave-2dw5 krb5kdc[22022](info): set up 2 sockets
Sep 26 13:13:12 dist-test-slave-2dw5 krb5kdc[22022](info): commencing operation
krb5kdc: starting...
W20260926 13:13:13.378707 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:13 dist-test-slave-2dw5 krb5kdc[22022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:13:13.496225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34315
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:13.578621 22035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:13.578907 22035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:13.578945 22035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:13.580194 22035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:13.580245 22035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:13.580262 22035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:13.580277 22035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:13.580292 22035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:13.581852 22035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:13.582103 22035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:13.582312 22035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:13.587132 22041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:13.587163 22043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:13.587131 22040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:13.587446 22035 server_base.cc:1034] running on GCE node
I20260926 13:13:13.587708 22035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:13.587994 22035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:13.589141 22035 hybrid_clock.cc:648] HybridClock initialized: now 1790428393589123 us; error 36 us; skew 500 ppm
Sep 26 13:13:13 dist-test-slave-2dw5 krb5kdc[22022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:13.591673 22035 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:13:13.592211 22035 webserver.cc:460] Webserver started at http://127.0.98.62:39717/ using document root <none> and password file <none>
I20260926 13:13:13.592425 22035 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:13.592473 22035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:13.592595 22035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:13.593665 22035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "f99153232f7248c9bb21e998b117d1cd"
format_stamp: "Formatted at 2026-09-26 13:13:13 on dist-test-slave-2dw5"
server_key: "92b1ffc4dee875631ce4cb1182428b08"
server_key_iv: "9eebf84b348816f087fc03cb42d9cb6b"
server_key_version: "encryptionkey@0"
I20260926 13:13:13.594003 22035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "f99153232f7248c9bb21e998b117d1cd"
format_stamp: "Formatted at 2026-09-26 13:13:13 on dist-test-slave-2dw5"
server_key: "92b1ffc4dee875631ce4cb1182428b08"
server_key_iv: "9eebf84b348816f087fc03cb42d9cb6b"
server_key_version: "encryptionkey@0"
I20260926 13:13:13.595387 22035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:13:13.596271 22050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:13.596464 22035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:13:13.596550 22035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "f99153232f7248c9bb21e998b117d1cd"
format_stamp: "Formatted at 2026-09-26 13:13:13 on dist-test-slave-2dw5"
server_key: "92b1ffc4dee875631ce4cb1182428b08"
server_key_iv: "9eebf84b348816f087fc03cb42d9cb6b"
server_key_version: "encryptionkey@0"
I20260926 13:13:13.596619 22035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:13.619632 22035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:13.621821 22035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:13.621965 22035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:13.625860 22035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34315
I20260926 13:13:13.625980 22102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34315 every 8 connection(s)
I20260926 13:13:13.626235 22035 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:13.627147 22103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:13.629642 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd: Bootstrap starting.
I20260926 13:13:13.630228 22103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:13.630451 22103 log.cc:824] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:13.631302 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd: No bootstrap required, opened a new log
I20260926 13:13:13.631328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 22035
I20260926 13:13:13.631448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:13.631625 392 external_mini_cluster.cc:1428] Setting key b89bd5eef4c25f4936cee13ba868a122
I20260926 13:13:13.633343 22103 raft_consensus.cc:348] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99153232f7248c9bb21e998b117d1cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34315 } }
I20260926 13:13:13.633493 22103 raft_consensus.cc:374] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:13.633525 22103 raft_consensus.cc:729] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f99153232f7248c9bb21e998b117d1cd, State: Initialized, Role: FOLLOWER
I20260926 13:13:13.633670 22103 consensus_queue.cc:260] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99153232f7248c9bb21e998b117d1cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34315 } }
I20260926 13:13:13.633754 22103 raft_consensus.cc:388] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:13.633787 22103 raft_consensus.cc:482] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:13.633837 22103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:13.634402 22103 raft_consensus.cc:504] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99153232f7248c9bb21e998b117d1cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34315 } }
I20260926 13:13:13.634490 22103 leader_election.cc:302] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f99153232f7248c9bb21e998b117d1cd; no voters:
I20260926 13:13:13.634637 22103 leader_election.cc:288] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:13.634673 22108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:13.634822 22108 raft_consensus.cc:686] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [term 1 LEADER]: Becoming Leader. State: Replica: f99153232f7248c9bb21e998b117d1cd, State: Running, Role: LEADER
I20260926 13:13:13.634918 22103 sys_catalog.cc:564] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:13.634953 22108 consensus_queue.cc:237] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99153232f7248c9bb21e998b117d1cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34315 } }
I20260926 13:13:13.635264 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f99153232f7248c9bb21e998b117d1cd. Latest consensus state: current_term: 1 leader_uuid: "f99153232f7248c9bb21e998b117d1cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99153232f7248c9bb21e998b117d1cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34315 } } }
I20260926 13:13:13.635459 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:13.635396 22109 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f99153232f7248c9bb21e998b117d1cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99153232f7248c9bb21e998b117d1cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34315 } } }
I20260926 13:13:13.635705 22109 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:13.636137 22117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:13.636618 22117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:13.638096 22117 catalog_manager.cc:1347] Generated new cluster ID: 1803f28a9e004785b640bcb2ac2b2b56
I20260926 13:13:13.638146 22117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:13.638494 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:13.632154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48791 (local address 127.0.98.62:34315)
0926 13:13:13.632309 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:13.632316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:13.632341 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:13:13.633322 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:13.633330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:13.633390 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:13.633546 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:13.633774 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:13.635327 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:13.636392 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:13.636935 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:13.637048 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:13.637820 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:13.637824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:13.637825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:13.637995 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:13.638037 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:13.638053 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:13.638127 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:13.638257 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 13:13:13.638288 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260926 13:13:13.639439 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:13.647064 22117 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:13.647766 22117 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:13.650157 22107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:13.643122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47299 (local address 127.0.98.62:34315)
0926 13:13:13.643160 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:13.643162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:13.646646 (+ 3484us) server_negotiation.cc:407] Connection header received
0926 13:13:13.646678 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:13.646679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:13.646691 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:13.646764 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:13.646979 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:13.647811 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:13.648815 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:13.649164 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:13.649236 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:13.649901 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:13.649902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:13.649903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:13.649907 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:13.649924 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:13.649929 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:13.649990 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:13.650053 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 13:13:13.650067 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:13:13.654441 22117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f99153232f7248c9bb21e998b117d1cd: Generated new TSK 0
I20260926 13:13:13.654644 22117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:13.664026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 22035
W20260926 13:13:13.667835 22020 connection.cc:535] client connection to 127.0.98.62:34315 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34315): Connection reset by peer (error 104)
2026-09-26T13:13:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:13 dist-test-slave-2dw5 krb5kdc[22134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:13 dist-test-slave-2dw5 krb5kdc[22134](info): set up 2 sockets
Sep 26 13:13:13 dist-test-slave-2dw5 krb5kdc[22134](info): commencing operation
krb5kdc: starting...
W20260926 13:13:14.610783 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:14 dist-test-slave-2dw5 krb5kdc[22134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:13:14.726133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46881
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:14.807848 22147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:14.808099 22147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:14.808133 22147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:14.809614 22147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:14.809697 22147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:14.809721 22147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:14.809739 22147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:14.809754 22147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:14.811571 22147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46881
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:14.811864 22147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:14.812095 22147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:14.817083 22153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:14.817179 22152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:14.817212 22155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:14.817441 22147 server_base.cc:1034] running on GCE node
I20260926 13:13:14.817723 22147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:14.818089 22147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:14.819281 22147 hybrid_clock.cc:648] HybridClock initialized: now 1790428394819234 us; error 61 us; skew 500 ppm
Sep 26 13:13:14 dist-test-slave-2dw5 krb5kdc[22134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:14.830646 22147 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:13:14.831229 22147 webserver.cc:460] Webserver started at http://127.0.98.62:33981/ using document root <none> and password file <none>
I20260926 13:13:14.831465 22147 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:14.831514 22147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:14.831653 22147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:14.832825 22147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "f3b67fbf0bee45e38898f234083696b8"
format_stamp: "Formatted at 2026-09-26 13:13:14 on dist-test-slave-2dw5"
server_key: "bf22ebfb0b392e1aff663a0774b8c6e6"
server_key_iv: "3a1a2b1fe1fd6c4b1b44e238e5833c3e"
server_key_version: "encryptionkey@0"
I20260926 13:13:14.833200 22147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3b67fbf0bee45e38898f234083696b8"
format_stamp: "Formatted at 2026-09-26 13:13:14 on dist-test-slave-2dw5"
server_key: "bf22ebfb0b392e1aff663a0774b8c6e6"
server_key_iv: "3a1a2b1fe1fd6c4b1b44e238e5833c3e"
server_key_version: "encryptionkey@0"
I20260926 13:13:14.834746 22147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:13:14.835875 22162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:14.836128 22147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:13:14.836237 22147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "f3b67fbf0bee45e38898f234083696b8"
format_stamp: "Formatted at 2026-09-26 13:13:14 on dist-test-slave-2dw5"
server_key: "bf22ebfb0b392e1aff663a0774b8c6e6"
server_key_iv: "3a1a2b1fe1fd6c4b1b44e238e5833c3e"
server_key_version: "encryptionkey@0"
I20260926 13:13:14.836313 22147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:14.839761 22147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:14.842162 22147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:14.842306 22147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:14.846129 22147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46881
I20260926 13:13:14.846166 22214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46881 every 8 connection(s)
I20260926 13:13:14.846534 22147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:14.847597 22215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:14.849828 22215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8: Bootstrap starting.
I20260926 13:13:14.850368 22215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:14.850626 22215 log.cc:824] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:14.851025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 22147
I20260926 13:13:14.851126 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:14.851293 392 external_mini_cluster.cc:1428] Setting key 9508c1d121130430d54c102d5e92eccc
I20260926 13:13:14.851432 22215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8: No bootstrap required, opened a new log
I20260926 13:13:14.853422 22215 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b67fbf0bee45e38898f234083696b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46881 } }
I20260926 13:13:14.853600 22215 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:14.853641 22215 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b67fbf0bee45e38898f234083696b8, State: Initialized, Role: FOLLOWER
I20260926 13:13:14.853796 22215 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b67fbf0bee45e38898f234083696b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46881 } }
I20260926 13:13:14.854108 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:14.851835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42399 (local address 127.0.98.62:46881)
0926 13:13:14.852030 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:14.852037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:14.852069 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:13:14.852691 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:14.852698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:14.852726 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:14.852865 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:14.853670 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:14.853674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:14.853674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:14.853678 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:14.853726 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:14.853741 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:14.853795 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:14.853856 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 13:13:14.853907 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260926 13:13:14.854127 22215 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:14.854167 22215 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:14.854210 22215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:14.855043 22215 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b67fbf0bee45e38898f234083696b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46881 } }
I20260926 13:13:14.855109 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:14.855155 22215 leader_election.cc:302] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3b67fbf0bee45e38898f234083696b8; no voters:
I20260926 13:13:14.855329 22215 leader_election.cc:288] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:14.855400 22220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:14.855541 22220 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [term 1 LEADER]: Becoming Leader. State: Replica: f3b67fbf0bee45e38898f234083696b8, State: Running, Role: LEADER
I20260926 13:13:14.855556 22215 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:14.855695 22220 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b67fbf0bee45e38898f234083696b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46881 } }
I20260926 13:13:14.856014 22220 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3b67fbf0bee45e38898f234083696b8. Latest consensus state: current_term: 1 leader_uuid: "f3b67fbf0bee45e38898f234083696b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b67fbf0bee45e38898f234083696b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46881 } } }
I20260926 13:13:14.856068 22222 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3b67fbf0bee45e38898f234083696b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b67fbf0bee45e38898f234083696b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46881 } } }
I20260926 13:13:14.856151 22220 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:14.856158 22222 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:14.856415 22232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:14.856914 22232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:14.858459 22232 catalog_manager.cc:1347] Generated new cluster ID: 85600c7df3084f90afd0c89faba81bd3
I20260926 13:13:14.858508 22232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:14.859112 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:14.857375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37413 (local address 127.0.98.62:46881)
0926 13:13:14.857413 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:14.857414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:14.858089 (+ 675us) server_negotiation.cc:407] Connection header received
0926 13:13:14.858141 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:14.858142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:14.858148 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:14.858195 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:14.858879 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:14.858881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:14.858881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:14.858886 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:14.858900 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:14.858904 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:14.858934 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:14.859039 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 13:13:14.859051 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:13:14.866912 22232 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:14.867094 22232 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:14.877351 22232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3b67fbf0bee45e38898f234083696b8: Generated new TSK 0
I20260926 13:13:14.877607 22232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:14.879165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 22147
2026-09-26T13:13:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:14 dist-test-slave-2dw5 krb5kdc[22246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:14 dist-test-slave-2dw5 krb5kdc[22246](info): set up 2 sockets
Sep 26 13:13:14 dist-test-slave-2dw5 krb5kdc[22246](info): commencing operation
krb5kdc: starting...
W20260926 13:13:15.826766 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:15 dist-test-slave-2dw5 krb5kdc[22246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:13:15.945540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41513
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:16.025996 22259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:16.026219 22259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:16.026244 22259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:16.027474 22259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:16.027523 22259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:16.027535 22259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:16.027546 22259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:16.027565 22259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:16.029084 22259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:16.029343 22259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:16.029558 22259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:16.034587 22264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:16.034577 22265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:16.034577 22267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:16.034812 22259 server_base.cc:1034] running on GCE node
I20260926 13:13:16.035153 22259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:16.035475 22259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:16.036634 22259 hybrid_clock.cc:648] HybridClock initialized: now 1790428396036618 us; error 28 us; skew 500 ppm
Sep 26 13:13:16 dist-test-slave-2dw5 krb5kdc[22246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:16.039143 22259 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:13:16.039590 22259 webserver.cc:460] Webserver started at http://127.0.98.62:34511/ using document root <none> and password file <none>
I20260926 13:13:16.039798 22259 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:16.039847 22259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:16.039970 22259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:16.041054 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "98f04db5f3f24520b8befe703b2a75f0"
format_stamp: "Formatted at 2026-09-26 13:13:16 on dist-test-slave-2dw5"
server_key: "30870b16f73a2e49f1822f363a39351e"
server_key_iv: "16620255977ffce30c618f14cb31c8bc"
server_key_version: "encryptionkey@0"
I20260926 13:13:16.041386 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "98f04db5f3f24520b8befe703b2a75f0"
format_stamp: "Formatted at 2026-09-26 13:13:16 on dist-test-slave-2dw5"
server_key: "30870b16f73a2e49f1822f363a39351e"
server_key_iv: "16620255977ffce30c618f14cb31c8bc"
server_key_version: "encryptionkey@0"
I20260926 13:13:16.042686 22259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:13:16.043401 22274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:16.043533 22259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:16.043613 22259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "98f04db5f3f24520b8befe703b2a75f0"
format_stamp: "Formatted at 2026-09-26 13:13:16 on dist-test-slave-2dw5"
server_key: "30870b16f73a2e49f1822f363a39351e"
server_key_iv: "16620255977ffce30c618f14cb31c8bc"
server_key_version: "encryptionkey@0"
I20260926 13:13:16.043699 22259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:16.053366 22259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:16.055502 22259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:16.055650 22259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:16.059403 22259 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41513
I20260926 13:13:16.059453 22326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41513 every 8 connection(s)
I20260926 13:13:16.059787 22259 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:16.060173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 22259
I20260926 13:13:16.060289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:16.060505 392 external_mini_cluster.cc:1428] Setting key 1aad213cdd100463dba8051c10131f34
I20260926 13:13:16.060801 22327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:16.064422 22327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0: Bootstrap starting.
I20260926 13:13:16.065153 22327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:16.065464 22327 log.cc:824] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:16.066269 22327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0: No bootstrap required, opened a new log
Sep 26 13:13:16 dist-test-slave-2dw5 krb5kdc[22246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:13:16.068528 22327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f04db5f3f24520b8befe703b2a75f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41513 } }
I20260926 13:13:16.068686 22327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:16.068707 22327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f04db5f3f24520b8befe703b2a75f0, State: Initialized, Role: FOLLOWER
I20260926 13:13:16.068866 22327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f04db5f3f24520b8befe703b2a75f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41513 } }
I20260926 13:13:16.068977 22327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:16.069010 22327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:16.069080 22327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:16.069793 22327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f04db5f3f24520b8befe703b2a75f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41513 } }
I20260926 13:13:16.069921 22327 leader_election.cc:302] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98f04db5f3f24520b8befe703b2a75f0; no voters:
I20260926 13:13:16.070101 22327 leader_election.cc:288] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:16.070149 22332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:16.070298 22332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [term 1 LEADER]: Becoming Leader. State: Replica: 98f04db5f3f24520b8befe703b2a75f0, State: Running, Role: LEADER
I20260926 13:13:16.070365 22327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:16.070423 22332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f04db5f3f24520b8befe703b2a75f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41513 } }
I20260926 13:13:16.070979 22333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98f04db5f3f24520b8befe703b2a75f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f04db5f3f24520b8befe703b2a75f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41513 } } }
I20260926 13:13:16.071059 22333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:16.071269 22334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98f04db5f3f24520b8befe703b2a75f0. Latest consensus state: current_term: 1 leader_uuid: "98f04db5f3f24520b8befe703b2a75f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f04db5f3f24520b8befe703b2a75f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41513 } } }
I20260926 13:13:16.071376 22334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:16.072089 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:16.061090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46291 (local address 127.0.98.62:41513)
0926 13:13:16.061249 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:16.061258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:16.061286 (+ 28us) server_negotiation.cc:407] Connection header received
0926 13:13:16.061957 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:16.061964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:16.062020 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:16.062141 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:16.063020 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:16.063945 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:16.065043 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:16.065502 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:16.065620 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:16.068220 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:16.068226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:16.068226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:16.068241 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:16.070221 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:16.070630 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:16.070633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:16.070633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:16.070731 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:16.070990 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:16.070992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:16.070992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:16.071383 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:16.071539 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:16.071797 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 13:13:16.071910 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
W20260926 13:13:16.072168 22348 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:13:16.072314 22348 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:13:16.072367 22341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:16.072536 22341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:16.073225 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:16.074100 22341 catalog_manager.cc:1347] Generated new cluster ID: e75e8d184e994d1bac8c6ec49cf715f0
I20260926 13:13:16.074153 22341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:16.081867 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:16.074666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59307 (local address 127.0.98.62:41513)
0926 13:13:16.074895 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:16.074896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:16.074990 (+ 94us) server_negotiation.cc:407] Connection header received
0926 13:13:16.075053 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:16.075054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:16.075063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:16.075113 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:16.075749 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:16.076456 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:16.077458 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:16.077782 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:16.077835 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:16.079270 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:16.079271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:16.079271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:16.079278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:16.080761 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:16.081158 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:16.081159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:16.081160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:16.081213 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:16.081388 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:16.081388 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:16.081389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:16.081504 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:16.081582 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:16.081704 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 13:13:16.081775 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260926 13:13:16.093995 22341 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:16.094806 22341 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:16.105417 22341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98f04db5f3f24520b8befe703b2a75f0: Generated new TSK 0
I20260926 13:13:16.105647 22341 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:16.112154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 22259
2026-09-26T13:13:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:16 dist-test-slave-2dw5 krb5kdc[22358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:16 dist-test-slave-2dw5 krb5kdc[22358](info): set up 2 sockets
Sep 26 13:13:16 dist-test-slave-2dw5 krb5kdc[22358](info): commencing operation
krb5kdc: starting...
W20260926 13:13:17.066869 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:17 dist-test-slave-2dw5 krb5kdc[22358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:13:17.187877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46275
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:17.267992 22371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:17.268226 22371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:17.268251 22371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:17.269418 22371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:17.269457 22371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:17.269470 22371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:17.269481 22371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:17.269500 22371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:17.271029 22371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:17.271239 22371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:17.271414 22371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:17.276453 22376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:17.276659 22371 server_base.cc:1034] running on GCE node
W20260926 13:13:17.276674 22379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:17.276674 22377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:17.277025 22371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:17.277308 22371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:17.278456 22371 hybrid_clock.cc:648] HybridClock initialized: now 1790428397278444 us; error 32 us; skew 500 ppm
Sep 26 13:13:17 dist-test-slave-2dw5 krb5kdc[22358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:17.281100 22371 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:13:17.281575 22371 webserver.cc:460] Webserver started at http://127.0.98.62:41635/ using document root <none> and password file <none>
I20260926 13:13:17.281765 22371 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:17.281803 22371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:17.281899 22371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:17.282998 22371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "b8e06a6726784f21b97890ac5bc6e52a"
format_stamp: "Formatted at 2026-09-26 13:13:17 on dist-test-slave-2dw5"
server_key: "12f354b070e8dc5f997cd99d5da4af66"
server_key_iv: "3b154ca734dac592c6e2f824416cb59c"
server_key_version: "encryptionkey@0"
I20260926 13:13:17.283306 22371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8e06a6726784f21b97890ac5bc6e52a"
format_stamp: "Formatted at 2026-09-26 13:13:17 on dist-test-slave-2dw5"
server_key: "12f354b070e8dc5f997cd99d5da4af66"
server_key_iv: "3b154ca734dac592c6e2f824416cb59c"
server_key_version: "encryptionkey@0"
I20260926 13:13:17.284596 22371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:13:17.285523 22386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:17.285712 22371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:13:17.285797 22371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "b8e06a6726784f21b97890ac5bc6e52a"
format_stamp: "Formatted at 2026-09-26 13:13:17 on dist-test-slave-2dw5"
server_key: "12f354b070e8dc5f997cd99d5da4af66"
server_key_iv: "3b154ca734dac592c6e2f824416cb59c"
server_key_version: "encryptionkey@0"
I20260926 13:13:17.285858 22371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:17.295974 22371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:17.298316 22371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:17.298435 22371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:17.302507 22371 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46275
I20260926 13:13:17.302546 22438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46275 every 8 connection(s)
I20260926 13:13:17.302925 22371 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:17.303920 22439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:17.306284 22439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a: Bootstrap starting.
I20260926 13:13:17.306828 22439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:17.307128 22439 log.cc:824] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:17.307837 22439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a: No bootstrap required, opened a new log
I20260926 13:13:17.309508 22439 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e06a6726784f21b97890ac5bc6e52a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } }
I20260926 13:13:17.309656 22439 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:17.309688 22439 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e06a6726784f21b97890ac5bc6e52a, State: Initialized, Role: FOLLOWER
I20260926 13:13:17.309823 22439 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e06a6726784f21b97890ac5bc6e52a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } }
I20260926 13:13:17.309901 22439 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:17.309931 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:17.309985 22439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:17.310505 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e06a6726784f21b97890ac5bc6e52a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } }
I20260926 13:13:17.310601 22439 leader_election.cc:302] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8e06a6726784f21b97890ac5bc6e52a; no voters:
I20260926 13:13:17.310771 22439 leader_election.cc:288] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:17.310820 22442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:17.311035 22439 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:17.311045 22442 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [term 1 LEADER]: Becoming Leader. State: Replica: b8e06a6726784f21b97890ac5bc6e52a, State: Running, Role: LEADER
I20260926 13:13:17.311201 22442 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e06a6726784f21b97890ac5bc6e52a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } }
I20260926 13:13:17.311793 22444 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8e06a6726784f21b97890ac5bc6e52a. Latest consensus state: current_term: 1 leader_uuid: "b8e06a6726784f21b97890ac5bc6e52a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e06a6726784f21b97890ac5bc6e52a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } }
I20260926 13:13:17.311797 22443 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8e06a6726784f21b97890ac5bc6e52a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e06a6726784f21b97890ac5bc6e52a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46275 } } }
I20260926 13:13:17.311879 22444 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:17.311893 22443 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:17.312062 22451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:17.312523 22451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:17.312749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 22371
I20260926 13:13:17.312860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:17.313031 392 external_mini_cluster.cc:1428] Setting key 38d97e9a5ac2f675b356f3b7778e854c
I20260926 13:13:17.314088 22451 catalog_manager.cc:1347] Generated new cluster ID: c488738066624741bbcf501dda145a9d
I20260926 13:13:17.314159 22451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:17.319306 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:17.313492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60099 (local address 127.0.98.62:46275)
0926 13:13:17.313743 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:17.313761 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:17.313797 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:13:17.313915 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:17.313922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:17.313976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:17.314123 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:17.314338 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:17.315861 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:17.316938 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:17.317604 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:17.317713 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:17.318391 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:17.318395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:17.318396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:17.318623 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:17.318663 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:17.318679 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:17.318747 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:17.318806 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 13:13:17.318899 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260926 13:13:17.320225 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:17.325033 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:17.321627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60833 (local address 127.0.98.62:46275)
0926 13:13:17.321660 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:17.321662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:17.321794 (+ 132us) server_negotiation.cc:407] Connection header received
0926 13:13:17.321842 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:17.321842 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:17.321852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:17.321902 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:17.322034 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:17.322741 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:17.323788 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:17.324086 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:17.324138 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:17.324804 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:17.324806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:17.324806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:17.324812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:17.324828 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:17.324832 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:17.324879 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:17.324929 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 13:13:17.324954 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:13:17.348006 22451 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:17.348806 22451 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:17.363410 22451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8e06a6726784f21b97890ac5bc6e52a: Generated new TSK 0
I20260926 13:13:17.363724 22451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:17.368762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 22371
2026-09-26T13:13:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:17 dist-test-slave-2dw5 krb5kdc[22470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:17 dist-test-slave-2dw5 krb5kdc[22470](info): set up 2 sockets
Sep 26 13:13:17 dist-test-slave-2dw5 krb5kdc[22470](info): commencing operation
krb5kdc: starting...
W20260926 13:13:18.326892 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:18 dist-test-slave-2dw5 krb5kdc[22470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:13:18.447165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39813
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:18.527215 22483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:18.527448 22483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:18.527482 22483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:18.528713 22483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:18.528769 22483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:18.528787 22483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:18.528805 22483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:18.528820 22483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:18.530395 22483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:18.530664 22483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:18.530910 22483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:18.539487 22491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:18.540647 22489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:18.541306 22488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:18.541777 22483 server_base.cc:1034] running on GCE node
I20260926 13:13:18.542094 22483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:18.542412 22483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:18.543962 22483 hybrid_clock.cc:648] HybridClock initialized: now 1790428398543938 us; error 37 us; skew 500 ppm
Sep 26 13:13:18 dist-test-slave-2dw5 krb5kdc[22470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:18.546916 22483 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:13:18.547410 22483 webserver.cc:460] Webserver started at http://127.0.98.62:36911/ using document root <none> and password file <none>
I20260926 13:13:18.547597 22483 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:18.547638 22483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:18.547726 22483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:18.548787 22483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "38d103405db4490a8e58d5a936d2232b"
format_stamp: "Formatted at 2026-09-26 13:13:18 on dist-test-slave-2dw5"
server_key: "0a0586b9d713e8aeda383e0a4228483d"
server_key_iv: "2f7fb8eec9555e626b2bf19fb03ec8eb"
server_key_version: "encryptionkey@0"
I20260926 13:13:18.549101 22483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "38d103405db4490a8e58d5a936d2232b"
format_stamp: "Formatted at 2026-09-26 13:13:18 on dist-test-slave-2dw5"
server_key: "0a0586b9d713e8aeda383e0a4228483d"
server_key_iv: "2f7fb8eec9555e626b2bf19fb03ec8eb"
server_key_version: "encryptionkey@0"
I20260926 13:13:18.550596 22483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:13:18.551548 22498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:18.551923 22483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:13:18.552017 22483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "38d103405db4490a8e58d5a936d2232b"
format_stamp: "Formatted at 2026-09-26 13:13:18 on dist-test-slave-2dw5"
server_key: "0a0586b9d713e8aeda383e0a4228483d"
server_key_iv: "2f7fb8eec9555e626b2bf19fb03ec8eb"
server_key_version: "encryptionkey@0"
I20260926 13:13:18.552099 22483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:18.555536 22483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:18.557621 22483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:18.557725 22483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:18.561525 22483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39813
I20260926 13:13:18.561741 22550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39813 every 8 connection(s)
I20260926 13:13:18.561918 22483 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:18.562106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 22483
I20260926 13:13:18.562206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:18.562377 392 external_mini_cluster.cc:1428] Setting key 202fac93fd39c284f012142068026217
I20260926 13:13:18.563071 22551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:18.565007 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:18.562900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33569 (local address 127.0.98.62:39813)
0926 13:13:18.563115 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:18.563124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:18.563155 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:13:18.563716 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:18.563725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:18.563759 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:18.563918 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:18.564543 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:18.564546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:18.564547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:18.564551 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:18.564596 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:18.564611 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:18.564685 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:18.564776 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 13:13:18.564817 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":103,"threads_started":1}
I20260926 13:13:18.566090 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:18.566134 22551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b: Bootstrap starting.
I20260926 13:13:18.566709 22551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:18.567026 22551 log.cc:824] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:18.567785 22551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b: No bootstrap required, opened a new log
I20260926 13:13:18.568681 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:18.567632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48227 (local address 127.0.98.62:39813)
0926 13:13:18.567674 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:18.567676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:18.567894 (+ 218us) server_negotiation.cc:407] Connection header received
0926 13:13:18.567918 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:18.567919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:18.567924 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:18.567965 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:18.568497 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:18.568498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:18.568498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:13:18.568501 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:18.568515 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:13:18.568519 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:18.568559 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:18.568610 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 13:13:18.568621 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:13:18.570003 22551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d103405db4490a8e58d5a936d2232b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } }
I20260926 13:13:18.570152 22551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:13:18.570183 22551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38d103405db4490a8e58d5a936d2232b, State: Initialized, Role: FOLLOWER
I20260926 13:13:18.570302 22551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d103405db4490a8e58d5a936d2232b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } }
I20260926 13:13:18.570377 22551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:13:18.570408 22551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:13:18.570456 22551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:18.571133 22551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d103405db4490a8e58d5a936d2232b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } }
I20260926 13:13:18.571234 22551 leader_election.cc:302] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38d103405db4490a8e58d5a936d2232b; no voters:
I20260926 13:13:18.571427 22551 leader_election.cc:288] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:13:18.571501 22561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:18.571676 22561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [term 1 LEADER]: Becoming Leader. State: Replica: 38d103405db4490a8e58d5a936d2232b, State: Running, Role: LEADER
I20260926 13:13:18.571679 22551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:18.571780 22561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d103405db4490a8e58d5a936d2232b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } }
I20260926 13:13:18.572190 22562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38d103405db4490a8e58d5a936d2232b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d103405db4490a8e58d5a936d2232b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } }
I20260926 13:13:18.572279 22562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:18.572446 22563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38d103405db4490a8e58d5a936d2232b. Latest consensus state: current_term: 1 leader_uuid: "38d103405db4490a8e58d5a936d2232b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38d103405db4490a8e58d5a936d2232b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39813 } } }
I20260926 13:13:18.572512 22563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:18.572799 22570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:18.573276 22570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:18.574831 22570 catalog_manager.cc:1347] Generated new cluster ID: a555d61ac7e34be8b1f2e51f3c6cae56
I20260926 13:13:18.574976 22570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:18.586804 22570 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:18.587028 22570 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:18.594532 22570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38d103405db4490a8e58d5a936d2232b: Generated new TSK 0
I20260926 13:13:18.594756 22570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:13:18.604852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 22483
2026-09-26T13:13:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1236 ms)
[----------] 18 tests from AuthTokenIssuingTest (22333 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:18 dist-test-slave-2dw5 krb5kdc[22582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:18 dist-test-slave-2dw5 krb5kdc[22582](info): set up 2 sockets
Sep 26 13:13:18 dist-test-slave-2dw5 krb5kdc[22582](info): commencing operation
krb5kdc: starting...
W20260926 13:13:19.572083 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:19 dist-test-slave-2dw5 krb5kdc[22582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:13:19.694320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35289
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35289,127.0.98.61:35437,127.0.98.60:36703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:19.775902 22595 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:13:19.776144 22595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:19.776170 22595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:19.776199 22595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:19.777431 22595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:19.777475 22595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:19.777494 22595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:19.777520 22595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:19.777544 22595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:19.779146 22595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35289,127.0.98.61:35437,127.0.98.60:36703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:19.779397 22595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:19.779606 22595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:19.784425 22601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:19.784440 22600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:19.784435 22603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:19.784664 22595 server_base.cc:1034] running on GCE node
I20260926 13:13:19.784926 22595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:19.785193 22595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:19.786350 22595 hybrid_clock.cc:648] HybridClock initialized: now 1790428399786336 us; error 28 us; skew 500 ppm
Sep 26 13:13:19 dist-test-slave-2dw5 krb5kdc[22582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:19.789048 22595 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:13:19.789482 22595 webserver.cc:460] Webserver started at http://127.0.98.62:35871/ using document root <none> and password file <none>
I20260926 13:13:19.789659 22595 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:19.789695 22595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:19.789786 22595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:19.790897 22595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "1a62b63b81174e4f9b01f39e70c17ef3"
format_stamp: "Formatted at 2026-09-26 13:13:19 on dist-test-slave-2dw5"
server_key: "1b76da7f7f23724d6c7d0c110a47a33d"
server_key_iv: "e0ed519d9a650b86ea48dad643f7ade9"
server_key_version: "encryptionkey@0"
I20260926 13:13:19.791225 22595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a62b63b81174e4f9b01f39e70c17ef3"
format_stamp: "Formatted at 2026-09-26 13:13:19 on dist-test-slave-2dw5"
server_key: "1b76da7f7f23724d6c7d0c110a47a33d"
server_key_iv: "e0ed519d9a650b86ea48dad643f7ade9"
server_key_version: "encryptionkey@0"
I20260926 13:13:19.792555 22595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:13:19.793460 22610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:19.793668 22595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:13:19.793753 22595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "1a62b63b81174e4f9b01f39e70c17ef3"
format_stamp: "Formatted at 2026-09-26 13:13:19 on dist-test-slave-2dw5"
server_key: "1b76da7f7f23724d6c7d0c110a47a33d"
server_key_iv: "e0ed519d9a650b86ea48dad643f7ade9"
server_key_version: "encryptionkey@0"
I20260926 13:13:19.793815 22595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:19.802891 22595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:19.804951 22595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:19.805078 22595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:19.808713 22595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35289
I20260926 13:13:19.808749 22662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35289 every 8 connection(s)
I20260926 13:13:19.809072 22595 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:19.809231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 22595
I20260926 13:13:19.809345 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:19.809536 392 external_mini_cluster.cc:1428] Setting key 315cf055550958674657263b206d8917
I20260926 13:13:19.810223 22663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:19.812027 22663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 13:13:19 dist-test-slave-2dw5 krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:13:19.822822 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.812667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35289 (local address 127.0.0.1:33442)
0926 13:13:19.812904 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:19.812941 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:19.813434 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:19.813705 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:19.813707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:19.813983 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:19.814144 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:19.814148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:19.815212 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:19.815212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:19.815895 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:19.815897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:19.816386 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:19.816387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:19.816447 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:19.816883 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:19.816891 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:19.819591 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:19.821335 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:19.821342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:19.821344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:19.821538 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:19.821787 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:19.821788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:19.821788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:19.821864 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:19.822308 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:19.822314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:19.822383 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 13:13:19.822467 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 13:13:19.822612 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260926 13:13:19.822831 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.812707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33442 (local address 127.0.98.62:35289)
0926 13:13:19.812869 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:19.812879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:19.813028 (+ 149us) server_negotiation.cc:407] Connection header received
0926 13:13:19.813592 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:19.813593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:19.813614 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:19.813694 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:19.814241 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:19.815130 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:19.815970 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:19.816297 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:19.816393 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:19.819673 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:19.819680 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:19.819680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:19.819687 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:19.821248 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:19.821614 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:19.821617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:19.821618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:19.821726 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:19.821916 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:19.821917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:19.821917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:19.822223 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:19.822310 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:19.822528 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 13:13:19.822705 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260926 13:13:19.823602 22663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:13:19.823920 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.823821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35437 (local address 127.0.0.1:42256)
0926 13:13:19.823862 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:19.823902 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 13:13:19.824043 22612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35437: connect: Connection refused (error 111)
I20260926 13:13:19.824422 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.824342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35437 (local address 127.0.0.1:42264)
0926 13:13:19.824381 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:19.824403 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 13:13:19.824523 22663 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35437: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35437: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 13:13:19.834208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35437
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35289,127.0.98.61:35437,127.0.98.60:36703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:13:19.873775 22663 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } attempt: 1
I20260926 13:13:19.874466 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.874263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35437 (local address 127.0.0.1:42278)
0926 13:13:19.874356 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:19.874423 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 13:13:19.874895 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.874761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35437 (local address 127.0.0.1:42282)
0926 13:13:19.874802 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:19.874834 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 13:13:19.874992 22663 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35437: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35437: connect: Connection refused (error 111)
W20260926 13:13:19.917332 22670 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:13:19.917578 22670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:19.917598 22670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:19.917619 22670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:19.918919 22670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:19.918974 22670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:19.918990 22670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:19.919004 22670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:19.919018 22670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:19.920609 22670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35289,127.0.98.61:35437,127.0.98.60:36703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:19.920861 22670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:19.921100 22670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:13:19.926354 22670 server_base.cc:1034] running on GCE node
W20260926 13:13:19.926333 22676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:19.926337 22678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:19.926337 22675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:19.926931 22670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:19.927202 22670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:19.928361 22670 hybrid_clock.cc:648] HybridClock initialized: now 1790428399928347 us; error 29 us; skew 500 ppm
Sep 26 13:13:19 dist-test-slave-2dw5 krb5kdc[22582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:19.935506 22670 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 13:13:19.936038 22670 webserver.cc:460] Webserver started at http://127.0.98.61:37073/ using document root <none> and password file <none>
I20260926 13:13:19.936241 22670 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:19.936290 22670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:19.936414 22670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:19.937528 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/data/instance:
uuid: "a5e5db87890e450eadc6ee4900041bde"
format_stamp: "Formatted at 2026-09-26 13:13:19 on dist-test-slave-2dw5"
server_key: "1519855f09716a2bc2f2976c241ef20a"
server_key_iv: "7fbbb0d5d5b256946dc0b5a33ee57eca"
server_key_version: "encryptionkey@0"
I20260926 13:13:19.937870 22670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/wal/instance:
uuid: "a5e5db87890e450eadc6ee4900041bde"
format_stamp: "Formatted at 2026-09-26 13:13:19 on dist-test-slave-2dw5"
server_key: "1519855f09716a2bc2f2976c241ef20a"
server_key_iv: "7fbbb0d5d5b256946dc0b5a33ee57eca"
server_key_version: "encryptionkey@0"
I20260926 13:13:19.939266 22670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:13:19.940040 22685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:19.940220 22670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:13:19.940308 22670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/wal
uuid: "a5e5db87890e450eadc6ee4900041bde"
format_stamp: "Formatted at 2026-09-26 13:13:19 on dist-test-slave-2dw5"
server_key: "1519855f09716a2bc2f2976c241ef20a"
server_key_iv: "7fbbb0d5d5b256946dc0b5a33ee57eca"
server_key_version: "encryptionkey@0"
I20260926 13:13:19.940371 22670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:19.943116 22663 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } attempt: 2
I20260926 13:13:19.943521 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.943382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35437 (local address 127.0.0.1:42284)
0926 13:13:19.943440 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:19.943484 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 13:13:19.943822 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.943735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35437 (local address 127.0.0.1:42298)
0926 13:13:19.943768 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:19.943798 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 13:13:19.943917 22663 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35437: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35437: connect: Connection refused (error 111)
I20260926 13:13:19.956503 22670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:19.958731 22670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:19.958920 22670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:19.962802 22670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35437
I20260926 13:13:19.962879 22737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35437 every 8 connection(s)
I20260926 13:13:19.963229 22670 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
I20260926 13:13:19.964327 22738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:19.965914 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:13:19.969014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 22670
I20260926 13:13:19.969106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-1/wal/instance
I20260926 13:13:19.969252 392 external_mini_cluster.cc:1428] Setting key 3f33af75235b4001e8d8bd460e34d820
Sep 26 13:13:19 dist-test-slave-2dw5 krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:13:19.975176 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.966679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33452 (local address 127.0.98.62:35289)
0926 13:13:19.966769 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:19.966771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:19.966971 (+ 200us) server_negotiation.cc:407] Connection header received
0926 13:13:19.967488 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:19.967489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:19.967512 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:19.967568 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:19.968164 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:19.968671 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:19.969794 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:19.970002 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:19.970117 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:19.972334 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:19.972335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:19.972335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:19.972342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:19.973631 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:19.974136 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:19.974138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:19.974138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:19.974209 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:19.974501 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:19.974502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:19.974502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:19.974620 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:19.974672 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:19.975034 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 13:13:19.975110 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:13:19.975407 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.966636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35289 (local address 127.0.0.1:33452)
0926 13:13:19.966827 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:19.966902 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:19.967380 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:19.967631 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:19.967635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:19.967884 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:19.968080 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:19.968084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:19.968771 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:19.968772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:19.969707 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:19.969710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:19.970090 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:19.970091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:19.970232 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:19.970723 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:19.970732 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:19.972234 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:19.973716 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:19.973726 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:19.973728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:19.974032 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:19.974280 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:19.974280 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:19.974281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:19.974424 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:19.974682 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:19.974687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:19.974882 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 13:13:19.974999 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 13:13:19.975170 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 13:13:19.975956 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 13:13:19 dist-test-slave-2dw5 krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:13:19.983439 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.976369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.0.98.61:35437)
0926 13:13:19.976547 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:19.976550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:19.976568 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:13:19.976586 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:19.976587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:19.976599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:19.976674 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:19.976919 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:19.977657 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:19.978283 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:19.978644 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:19.978702 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:19.980673 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:19.980681 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:19.980681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:19.980687 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:19.982158 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:19.982571 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:19.982576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:19.982576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:19.982646 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:19.982859 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:19.982860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:19.982860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:19.983058 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:19.983129 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:19.983280 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 13:13:19.983359 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260926 13:13:19.983577 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.976211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35437 (local address 127.0.0.1:42312)
0926 13:13:19.976265 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:19.976276 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:19.976320 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:19.976677 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:19.976678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:19.976783 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:19.976837 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:19.976838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:19.977728 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:19.977729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:19.978219 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:19.978221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:19.978712 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:19.978713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:19.978788 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:19.979267 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:19.979272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:19.980579 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:19.982244 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:19.982245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:19.982246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:19.982459 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:19.982716 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:19.982716 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:19.982717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:19.982779 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:19.983125 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:19.983126 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:19.983176 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 13:13:19.983255 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 13:13:19.983434 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":72}
I20260926 13:13:19.984266 22738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:13:19.984616 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.984478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58370)
0926 13:13:19.984533 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:19.984588 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 13:13:19.984759 22687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
I20260926 13:13:19.985174 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:19.985072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58384)
0926 13:13:19.985122 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:19.985150 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 13:13:19.985283 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36703: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 13:13:19.994294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36703
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35289,127.0.98.61:35437,127.0.98.60:36703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:13:20.034523 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } attempt: 1
I20260926 13:13:20.035090 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.034924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58394)
0926 13:13:20.034991 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.035048 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 13:13:20.035064 22663 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } attempt: 3
I20260926 13:13:20.035451 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.035359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58404)
0926 13:13:20.035398 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.035432 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 13:13:20.035579 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36703: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
Sep 26 13:13:20 dist-test-slave-2dw5 krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:13:20.042486 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.035361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.0.98.61:35437)
0926 13:13:20.035420 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.035422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.035438 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:13:20.035462 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.035463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.035474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.035863 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.035895 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.036371 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.037428 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.037655 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.037725 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.039940 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:20.039941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:20.039941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:20.039951 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:20.041427 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.041792 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.041793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.041793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.041845 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.042014 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.042014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.042014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.042159 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:20.042206 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:20.042341 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 13:13:20.042418 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:13:20.042547 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.035317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35437 (local address 127.0.0.1:42316)
0926 13:13:20.035365 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.035376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:20.035417 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:20.035554 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:20.035555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:20.035730 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:20.035805 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:20.035807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.036490 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.036492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:20.037306 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:20.037310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.037734 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.037735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:20.037818 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.038273 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:20.038278 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:20.039789 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:20.041508 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:20.041509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:20.041510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:20.041721 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:20.041901 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:20.041902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:20.041902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:20.041964 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:20.042209 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:20.042210 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:20.042259 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 13:13:20.042322 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:13:20.042437 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 13:13:20.043020 22663 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:13:20.043354 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.043254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58420)
0926 13:13:20.043300 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.043332 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 13:13:20.043610 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.043540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58436)
0926 13:13:20.043574 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.043595 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 13:13:20.043694 22663 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36703: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
I20260926 13:13:20.074911 22663 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } attempt: 1
I20260926 13:13:20.075374 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.075222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58448)
0926 13:13:20.075289 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.075338 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 13:13:20.075721 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.075633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58460)
0926 13:13:20.075671 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.075699 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 13:13:20.075807 22663 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36703: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
W20260926 13:13:20.078320 22745 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:13:20.078526 22745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:20.078543 22745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:20.078567 22745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:20.079818 22745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:20.079861 22745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:20.079874 22745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:20.079885 22745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:20.079905 22745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:20.081449 22745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35289,127.0.98.61:35437,127.0.98.60:36703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:20.081700 22745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:20.081945 22745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:20.086951 22753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:20.086978 22745 server_base.cc:1034] running on GCE node
W20260926 13:13:20.086951 22751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:20.087090 22750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:20.087476 22745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:20.087772 22745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:20.088945 22745 hybrid_clock.cc:648] HybridClock initialized: now 1790428400088920 us; error 43 us; skew 500 ppm
Sep 26 13:13:20 dist-test-slave-2dw5 krb5kdc[22582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:20.091635 22745 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 13:13:20.092146 22745 webserver.cc:460] Webserver started at http://127.0.98.60:40181/ using document root <none> and password file <none>
I20260926 13:13:20.092368 22745 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:20.092424 22745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:20.092546 22745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:20.093699 22745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/data/instance:
uuid: "faefc175b01241e7a46753ce467bdefb"
format_stamp: "Formatted at 2026-09-26 13:13:20 on dist-test-slave-2dw5"
server_key: "bea633ce7af66507e5730de7b29b82d7"
server_key_iv: "31bf4aaa84e53286a7ecc6177267dd91"
server_key_version: "encryptionkey@0"
I20260926 13:13:20.094023 22745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/wal/instance:
uuid: "faefc175b01241e7a46753ce467bdefb"
format_stamp: "Formatted at 2026-09-26 13:13:20 on dist-test-slave-2dw5"
server_key: "bea633ce7af66507e5730de7b29b82d7"
server_key_iv: "31bf4aaa84e53286a7ecc6177267dd91"
server_key_version: "encryptionkey@0"
I20260926 13:13:20.095321 22745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:13:20.096118 22760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:20.096259 22745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:13:20.096372 22745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/wal
uuid: "faefc175b01241e7a46753ce467bdefb"
format_stamp: "Formatted at 2026-09-26 13:13:20 on dist-test-slave-2dw5"
server_key: "bea633ce7af66507e5730de7b29b82d7"
server_key_iv: "31bf4aaa84e53286a7ecc6177267dd91"
server_key_version: "encryptionkey@0"
I20260926 13:13:20.096436 22745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:20.103714 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } attempt: 2
I20260926 13:13:20.104163 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.104045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58466)
0926 13:13:20.104101 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.104131 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 13:13:20.104503 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.104379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58468)
0926 13:13:20.104414 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.104437 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 13:13:20.104633 22738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36703: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36703: connect: Connection refused (error 111)
I20260926 13:13:20.119262 22745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:20.121488 22745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:20.121627 22745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:20.125465 22745 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36703
I20260926 13:13:20.125504 22812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36703 every 8 connection(s)
I20260926 13:13:20.125814 22745 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
I20260926 13:13:20.126766 22813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:20.128319 22813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:13:20.129650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 22745
I20260926 13:13:20.129732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428136114168-392-0/minicluster-data/master-2/wal/instance
I20260926 13:13:20.129891 392 external_mini_cluster.cc:1428] Setting key 948c19e450dc4f2dcf5927cd98b1a8fd
Sep 26 13:13:20 dist-test-slave-2dw5 krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 13:13:20 dist-test-slave-2dw5 krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:13:20.138325 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.129028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.0.98.62:35289)
0926 13:13:20.129107 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.129109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.129251 (+ 142us) server_negotiation.cc:407] Connection header received
0926 13:13:20.129740 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.129742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.129754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.129817 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.130584 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.131218 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.132613 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.132797 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.132861 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.135363 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:20.135365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:20.135365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:20.135372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:20.136822 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.137319 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.137320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.137321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.137382 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.137627 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.137628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.137628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.137768 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:20.137837 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:20.138154 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 13:13:20.138232 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:13:20.138476 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.128954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35289 (local address 127.0.0.1:33464)
0926 13:13:20.129120 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.129159 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:20.129633 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:20.129877 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:20.129880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:20.130217 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:20.130508 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:20.130513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.131345 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.131347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:20.132520 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:20.132522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.132897 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.132898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:20.133039 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.133602 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:20.133611 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:20.135268 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:20.136901 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:20.136909 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:20.136911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:20.137250 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:20.137437 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:20.137438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:20.137438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:20.137564 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:20.137839 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:20.137844 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:20.138019 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 13:13:20.138112 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 13:13:20.138240 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260926 13:13:20.139503 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.130248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58484 (local address 127.0.98.60:36703)
0926 13:13:20.130401 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.130406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.130443 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:13:20.130487 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.130489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.130507 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.130576 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.131741 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.132781 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.133803 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.134310 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.134366 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.136618 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:20.136627 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:20.136628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:20.136635 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:20.138176 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.138596 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.138599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.138599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.138653 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.138873 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.138874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.138874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.139053 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:20.139203 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:20.139249 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 13:13:20.139338 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260926 13:13:20.139693 22813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:13:20 dist-test-slave-2dw5 krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 13:13:20 dist-test-slave-2dw5 krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:13:20.146075 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.139919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35437 (local address 127.0.0.1:42326)
0926 13:13:20.139961 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.139973 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:20.140015 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:20.140214 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:20.140215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:20.140360 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:20.140420 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:20.140422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.141021 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.141022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:20.141524 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:20.141525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.141855 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.141856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:20.141927 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.142354 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:20.142359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:20.143763 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:20.145169 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:20.145170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:20.145171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:20.145327 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:20.145503 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:20.145504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:20.145505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:20.145561 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:20.145797 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:20.145798 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:20.145842 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 13:13:20.145894 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 13:13:20.145976 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 13:13:20.146098 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.140071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.0.98.61:35437)
0926 13:13:20.140115 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.140117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.140124 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:13:20.140148 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.140149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.140158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.140202 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.140477 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.140948 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.141600 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.141801 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.141854 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.143846 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:20.143847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:20.143848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:20.143855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:20.145092 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.145389 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.145390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.145391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.145448 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.145611 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.145612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.145612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.145725 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:20.145784 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:20.145933 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 13:13:20.146016 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:13:20.146610 22813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:13:20.146811 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.140242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.0.98.62:35289)
0926 13:13:20.140284 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.140285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.140314 (+ 29us) server_negotiation.cc:407] Connection header received
0926 13:13:20.140355 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.140356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.140364 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.140419 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.140854 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.141308 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.141920 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.142141 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.142206 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.144412 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:20.144414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:20.144414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:20.144420 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:20.145690 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.146011 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.146012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.146013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.146067 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.146241 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.146242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.146242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.146390 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:20.146569 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:20.146646 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 13:13:20.146727 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 13:13:20 dist-test-slave-2dw5 krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:13:20.150936 22663 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } attempt: 2
Sep 26 13:13:20 dist-test-slave-2dw5 krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:13:20.154019 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.146861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58488)
0926 13:13:20.146896 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.146906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:20.146945 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:20.147147 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:20.147148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:20.147287 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:20.147328 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:20.147329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.148206 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.148207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:20.148968 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:20.148970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.149475 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.149476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:20.149518 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.150046 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:20.150051 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:20.151260 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:20.153031 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:20.153032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:20.153032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:20.153190 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:20.153379 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:20.153380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:20.153380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:20.153448 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:20.153700 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:20.153700 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:20.153734 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 13:13:20.153797 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:13:20.153880 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 13:13:20.154093 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.147030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58488 (local address 127.0.98.60:36703)
0926 13:13:20.147065 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.147067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.147073 (+ 6us) server_negotiation.cc:407] Connection header received
0926 13:13:20.147088 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.147089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.147098 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.147139 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.147400 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.148119 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.149069 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.149383 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.149478 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.151338 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:20.151339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:20.151340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:20.151346 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:20.152956 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.153259 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.153260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.153260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.153306 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.153513 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.153514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.153514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.153621 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:20.153680 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:20.153866 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 13:13:20.153944 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:13:20.155426 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.147556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42328 (local address 127.0.98.61:35437)
0926 13:13:20.147589 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.147591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.147597 (+ 6us) server_negotiation.cc:407] Connection header received
0926 13:13:20.147615 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.147615 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.147623 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.147665 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.148206 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.148963 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.149882 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.150169 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.150246 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.152884 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:20.152885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:20.152886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:20.152893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:20.154370 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.154720 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.154721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.154721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.154775 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.154969 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.154970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.154970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.155078 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:20.155241 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:20.155275 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 13:13:20.155343 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 26 13:13:20 dist-test-slave-2dw5 krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:13:20.156570 22813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb: Bootstrap starting.
I20260926 13:13:20.156913 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:20.157275 22813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:20.157591 22813 log.cc:824] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:20.158414 22813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb: No bootstrap required, opened a new log
I20260926 13:13:20.157933 22637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1a62b63b81174e4f9b01f39e70c17ef3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33476
I20260926 13:13:20.159315 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.151182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58502)
0926 13:13:20.151232 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.151241 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:20.151275 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:20.152127 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:20.152129 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:20.152256 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:20.152304 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:20.152306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.153221 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.153222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:20.153968 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:20.153971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.154407 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.154409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:20.154463 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.154875 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:20.154880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:20.156397 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:20.158180 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:20.158181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:20.158181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:20.158345 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:20.158586 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:20.158587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:20.158587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:20.158650 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:20.158970 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:20.158971 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:20.159048 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 13:13:20.159125 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 13:13:20.159228 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 13:13:20.159469 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.151239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.0.98.60:36703)
0926 13:13:20.151987 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.151989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.151996 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:13:20.152013 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.152014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.152024 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.152377 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.152411 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.153134 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.154058 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.154309 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.154598 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.156468 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:20.156470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:20.156470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:20.156477 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:20.158104 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.158405 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.158406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.158406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.158507 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.158701 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.158702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.158702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.158831 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:20.158969 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:20.159169 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 13:13:20.159251 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":92,"threads_started":1}
I20260926 13:13:20.160513 22663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3: Bootstrap starting.
I20260926 13:13:20.160465 22813 raft_consensus.cc:348] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } }
I20260926 13:13:20.160614 22813 raft_consensus.cc:729] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faefc175b01241e7a46753ce467bdefb, State: Initialized, Role: FOLLOWER
I20260926 13:13:20.160697 22813 consensus_queue.cc:260] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } }
I20260926 13:13:20.161013 22813 sys_catalog.cc:564] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:20.160974 22820 sys_catalog.cc:455] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } } }
I20260926 13:13:20.161069 22820 sys_catalog.cc:458] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:20.161104 22663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:20.161345 22663 log.cc:824] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:20.162101 22663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3: No bootstrap required, opened a new log
W20260926 13:13:20.162786 22832 catalog_manager.cc:1558] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:13:20.162858 22832 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:13:20.163512 22663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } }
I20260926 13:13:20.163643 22663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a62b63b81174e4f9b01f39e70c17ef3, State: Initialized, Role: FOLLOWER
I20260926 13:13:20.163718 22663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } }
I20260926 13:13:20.164012 22663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:20.164057 22637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 13:13:20.164100 22637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:20.164870 22637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } }
I20260926 13:13:20.165117 22637 leader_election.cc:288] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [CANDIDATE]: Term 1 election: Requested vote from peers a5e5db87890e450eadc6ee4900041bde (127.0.98.61:35437), faefc175b01241e7a46753ce467bdefb (127.0.98.60:36703)
I20260926 13:13:20.163154 22745 master_runner.cc:186] Error getting master registration for 127.0.98.62:35289: OK, instance_id { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" instance_seqno: 1790428399807697 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:13:20.165906 22787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "faefc175b01241e7a46753ce467bdefb"
I20260926 13:13:20.165988 22745 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:13:20.166024 22787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:20.166941 22787 raft_consensus.cc:2454] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a62b63b81174e4f9b01f39e70c17ef3 in term 1.
I20260926 13:13:20.167205 22612 leader_election.cc:302] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a62b63b81174e4f9b01f39e70c17ef3, faefc175b01241e7a46753ce467bdefb; no voters:
I20260926 13:13:20.167419 22846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:20.167538 22846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [term 1 LEADER]: Becoming Leader. State: Replica: 1a62b63b81174e4f9b01f39e70c17ef3, State: Running, Role: LEADER
I20260926 13:13:20.167632 22846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } }
I20260926 13:13:20.164011 22834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } } }
I20260926 13:13:20.167991 22834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:20.168334 22834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a62b63b81174e4f9b01f39e70c17ef3. Latest consensus state: current_term: 1 leader_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } } }
I20260926 13:13:20.168529 22834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:20.166497 22595 master_runner.cc:186] Error getting master registration for 127.0.98.61:35437: OK, instance_id { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" instance_seqno: 1790428399961684 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:13:20.169282 22595 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 13:13:20.169672 22845 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:13:20.169725 22845 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:13:20.173409 22853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:20.173625 22853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:20.177562 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.169080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.0.98.61:35437)
0926 13:13:20.169164 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.169166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.169682 (+ 516us) server_negotiation.cc:407] Connection header received
0926 13:13:20.169728 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.169729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.169741 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.169788 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.170823 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.171575 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.172615 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.172912 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.172969 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.174607 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:20.174608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:20.174609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:20.174616 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:20.176156 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.176524 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.176525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.176525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.176591 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.176777 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.176778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.176778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.176901 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:20.177000 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:20.177385 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 13:13:20.177463 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:13:20.180033 22787 raft_consensus.cc:1261] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [term 1 FOLLOWER]: Refusing update from remote peer 1a62b63b81174e4f9b01f39e70c17ef3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:13:20.180392 22850 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:13:20.181891 22820 sys_catalog.cc:455] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a62b63b81174e4f9b01f39e70c17ef3. Latest consensus state: current_term: 1 leader_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } } }
I20260926 13:13:20.182003 22820 sys_catalog.cc:458] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:20.183007 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.169206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.0.98.62:35289)
0926 13:13:20.169262 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.169264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.170075 (+ 811us) server_negotiation.cc:407] Connection header received
0926 13:13:20.170221 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.170223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.170234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.170265 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.171529 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.172225 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.173849 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.174143 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.174185 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.179552 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:20.179554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:20.179555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:20.179562 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:20.181093 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.181467 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.181469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.181469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.181524 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.181987 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.181988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.181989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.182329 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:20.182422 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:20.182776 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 13:13:20.182905 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:13:20.184921 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.169624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.0.98.60:36703)
0926 13:13:20.169662 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.169664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.175061 (+ 5397us) server_negotiation.cc:407] Connection header received
0926 13:13:20.175135 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.175137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.175157 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.175188 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.175810 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.176527 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.178245 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.178625 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.178666 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.181304 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:20.181305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:20.181305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:20.181325 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:20.182900 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.183850 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.183851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.183851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.183922 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.184206 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.184207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.184207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.184343 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:20.184411 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:20.184732 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 13:13:20.184833 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:13:20.195765 22738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } attempt: 3
I20260926 13:13:20.196224 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } } }
I20260926 13:13:20.196324 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:20.196842 22820 sys_catalog.cc:455] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } } }
I20260926 13:13:20.196942 22820 sys_catalog.cc:458] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:20.197150 22858 mvcc.cc:205] Tried to move back new op lower bound from 7333594727134834688 to 7333594727089033216. Current Snapshot: MvccSnapshot[applied={T|T < 7333594727134834688}]
I20260926 13:13:20.198014 22853 catalog_manager.cc:1347] Generated new cluster ID: 49d2bcecbb3447a98e654e0cdfe73dcd
I20260926 13:13:20.198061 22853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:13:20.200630 22859 mvcc.cc:205] Tried to move back new op lower bound from 7333594727134834688 to 7333594727089033216. Current Snapshot: MvccSnapshot[applied={T|T < 7333594727134834688}]
Sep 26 13:13:20 dist-test-slave-2dw5 krb5kdc[22582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:13:20.205812 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.196184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36703 (local address 127.0.0.1:58522)
0926 13:13:20.196237 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:20.196249 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:20.196291 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:20.196572 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:20.196573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:20.196781 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:20.196832 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:20.196834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.198651 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.198652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:20.199649 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:20.199652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.200280 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.200281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:20.200356 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.200909 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:20.200926 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:20.202575 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:20.204497 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:20.204499 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:20.204499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:20.204746 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:20.205042 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:20.205042 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:20.205043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:20.205119 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:20.205459 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:20.205460 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:20.205526 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 13:13:20.205585 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:13:20.205701 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 13:13:20.207597 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:20.196217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58522 (local address 127.0.98.60:36703)
0926 13:13:20.196253 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.196255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.196278 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:13:20.196352 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.196353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.196362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.196404 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.197651 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.198424 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.199835 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:20.200149 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:20.200208 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:20.202699 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:20.202701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:20.202702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:20.202726 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:20.204408 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.204875 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.204877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.204877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.204957 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:20.205192 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:20.205193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:20.205193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:20.205348 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:20.205407 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:20.207363 (+ 1956us) server_negotiation.cc:299] Negotiation successful
0926 13:13:20.207494 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:13:20.209297 22738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde: Bootstrap starting.
I20260926 13:13:20.210070 22738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:20.210384 22738 log.cc:824] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:20.211465 22738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde: No bootstrap required, opened a new log
I20260926 13:13:20.213645 22738 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } }
I20260926 13:13:20.213804 22738 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e5db87890e450eadc6ee4900041bde, State: Initialized, Role: FOLLOWER
I20260926 13:13:20.213889 22738 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } }
I20260926 13:13:20.214249 22738 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:20.214372 22711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:20.215394 22711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [term 1 FOLLOWER]: Refusing update from remote peer 1a62b63b81174e4f9b01f39e70c17ef3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:13:20.215802 22834 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:13:20.216153 22712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a5e5db87890e450eadc6ee4900041bde"
I20260926 13:13:20.217346 22712 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a62b63b81174e4f9b01f39e70c17ef3 in term 1.
W20260926 13:13:20.220552 22875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:13:20.220679 22875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:13:20.222922 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } } }
I20260926 13:13:20.223032 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:20.223088 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a62b63b81174e4f9b01f39e70c17ef3. Latest consensus state: current_term: 1 leader_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } } }
I20260926 13:13:20.223133 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:20.223162 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } } }
I20260926 13:13:20.223202 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:20.226682 22877 mvcc.cc:205] Tried to move back new op lower bound from 7333594727134834688 to 7333594727089033216. Current Snapshot: MvccSnapshot[applied={T|T < 7333594727134834688}]
I20260926 13:13:20.250736 22834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a62b63b81174e4f9b01f39e70c17ef3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35289 } } peers { permanent_uuid: "a5e5db87890e450eadc6ee4900041bde" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35437 } } peers { permanent_uuid: "faefc175b01241e7a46753ce467bdefb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36703 } } }
I20260926 13:13:20.250886 22834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:20.252151 22853 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:20.252872 22853 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:20.262279 22853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a62b63b81174e4f9b01f39e70c17ef3: Generated new TSK 0
I20260926 13:13:20.262504 22853 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:13:20.286507 22667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.284734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.0.98.62:35289)
0926 13:13:20.284780 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.284782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.285778 (+ 996us) server_negotiation.cc:407] Connection header received
0926 13:13:20.285885 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.285886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.285898 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.285928 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.286430 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33484: BlockingRecv error: recv got EOF from 127.0.0.1:33484 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 13:13:20.287485 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.285136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.0.98.61:35437)
0926 13:13:20.285176 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.285178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.286939 (+ 1761us) server_negotiation.cc:407] Connection header received
0926 13:13:20.287035 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.287036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.287048 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.287077 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.287418 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42338: BlockingRecv error: recv got EOF from 127.0.0.1:42338 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 13:13:20.288347 22818 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.285496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.0.98.60:36703)
0926 13:13:20.285539 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.285540 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.287852 (+ 2312us) server_negotiation.cc:407] Connection header received
0926 13:13:20.287946 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.287947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.287958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.287986 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.288288 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58528: BlockingRecv error: recv got EOF from 127.0.0.1:58528 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 13:13:20.290542 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.289748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.0.98.61:35437)
0926 13:13:20.289797 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.289799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.290005 (+ 206us) server_negotiation.cc:407] Connection header received
0926 13:13:20.290114 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.290115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.290136 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.290166 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.290495 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42342: BlockingRecv error: recv got EOF from 127.0.0.1:42342 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 13:13:20.293859 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.292984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.0.98.61:35437)
0926 13:13:20.293021 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.293023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.293294 (+ 271us) server_negotiation.cc:407] Connection header received
0926 13:13:20.293408 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.293409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.293420 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.293450 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.293802 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42354: BlockingRecv error: recv got EOF from 127.0.0.1:42354 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 13:13:20.298194 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.297305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.0.98.61:35437)
0926 13:13:20.297345 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.297347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.297642 (+ 295us) server_negotiation.cc:407] Connection header received
0926 13:13:20.297740 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.297741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.297753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.297782 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.298138 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42362: BlockingRecv error: recv got EOF from 127.0.0.1:42362 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 13:13:20.304653 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.303698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42374 (local address 127.0.98.61:35437)
0926 13:13:20.303743 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.303745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.304025 (+ 280us) server_negotiation.cc:407] Connection header received
0926 13:13:20.304138 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.304139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.304151 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.304180 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.304576 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42374: BlockingRecv error: recv got EOF from 127.0.0.1:42374 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 13:13:20.315156 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.314174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.0.98.61:35437)
0926 13:13:20.314218 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.314220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.314499 (+ 279us) server_negotiation.cc:407] Connection header received
0926 13:13:20.314597 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.314598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.314610 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.314639 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.315092 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42376: BlockingRecv error: recv got EOF from 127.0.0.1:42376 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 13:13:20.333959 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.332840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42388 (local address 127.0.98.61:35437)
0926 13:13:20.332886 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.332888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.333211 (+ 323us) server_negotiation.cc:407] Connection header received
0926 13:13:20.333353 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.333354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.333389 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.333420 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.333894 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42388: BlockingRecv error: recv got EOF from 127.0.0.1:42388 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 13:13:20.368405 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.367511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42396 (local address 127.0.98.61:35437)
0926 13:13:20.367578 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.367581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.367706 (+ 125us) server_negotiation.cc:407] Connection header received
0926 13:13:20.367777 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.367778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.367788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.367828 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.368339 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42396: BlockingRecv error: recv got EOF from 127.0.0.1:42396 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 13:13:20.434293 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.433633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.0.98.61:35437)
0926 13:13:20.433706 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.433709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.433765 (+ 56us) server_negotiation.cc:407] Connection header received
0926 13:13:20.433841 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.433842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.433854 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.433902 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.434239 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42402: BlockingRecv error: recv got EOF from 127.0.0.1:42402 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 13:13:20.564184 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.563517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.0.98.61:35437)
0926 13:13:20.563602 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.563604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.563615 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:13:20.563698 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.563699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.563712 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.563776 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.564135 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42414: BlockingRecv error: recv got EOF from 127.0.0.1:42414 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 13:13:20.822046 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:20.821335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.0.98.61:35437)
0926 13:13:20.821411 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:20.821413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:20.821423 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:13:20.821505 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:20.821506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:20.821518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:20.821565 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:20.821998 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42422: BlockingRecv error: recv got EOF from 127.0.0.1:42422 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:13:21.163287 22832 catalog_manager.cc:1259] Loaded cluster ID: 49d2bcecbb3447a98e654e0cdfe73dcd
I20260926 13:13:21.163364 22832 catalog_manager.cc:1552] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb: loading cluster ID for follower catalog manager: success
I20260926 13:13:21.164228 22832 catalog_manager.cc:1574] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb: acquiring CA information for follower catalog manager: success
I20260926 13:13:21.164501 22832 catalog_manager.cc:1602] T 00000000000000000000000000000000 P faefc175b01241e7a46753ce467bdefb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:13:21.221261 22875 catalog_manager.cc:1259] Loaded cluster ID: 49d2bcecbb3447a98e654e0cdfe73dcd
I20260926 13:13:21.221350 22875 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde: loading cluster ID for follower catalog manager: success
I20260926 13:13:21.222158 22875 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde: acquiring CA information for follower catalog manager: success
I20260926 13:13:21.222466 22875 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5e5db87890e450eadc6ee4900041bde: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:13:21.338614 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:21.335554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.0.98.61:35437)
0926 13:13:21.335752 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:21.335755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:21.335767 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:13:21.335824 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:21.335825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:21.335912 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:21.335990 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:13:21.336154 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:21.336814 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:21.337683 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:21.337920 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:21.337981 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:21.338208 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:13:21.338367 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:13:21.338450 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:21.338471 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 13:13:21.338502 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260926 13:13:21.344949 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:21.341189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.0.98.60:36703)
0926 13:13:21.341314 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:21.341316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:21.341352 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:13:21.341467 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:21.341469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:21.341517 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:21.341599 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:13:21.341764 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:21.342568 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:21.343668 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:21.343965 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:21.344050 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:21.344270 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:13:21.344464 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:13:21.344559 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:21.344810 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 13:13:21.344844 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20260926 13:13:21.350700 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:21.348136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.0.98.60:36703)
0926 13:13:21.348212 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:21.348214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:21.348220 (+ 6us) server_negotiation.cc:407] Connection header received
0926 13:13:21.348243 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:21.348244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:21.348254 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:21.348301 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:13:21.348429 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:21.349015 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:21.349863 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:21.350073 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:21.350127 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:21.350272 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:13:21.350308 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:13:21.350355 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:21.350578 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 13:13:21.350628 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:13:21.351178 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:21.347809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.0.98.61:35437)
0926 13:13:21.347866 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:21.347868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:21.347881 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:13:21.347957 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:21.347958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:21.347974 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:21.348036 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:13:21.348267 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:21.349028 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:21.350249 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:21.350546 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:21.350606 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:21.350811 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:13:21.350895 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:13:21.350950 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:21.351046 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 13:13:21.351092 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 13:13:21.351656 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:21.347782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.0.98.62:35289)
0926 13:13:21.348383 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:21.348385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:21.348393 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:13:21.348447 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:21.348448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:21.348513 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:21.348604 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:13:21.348728 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:21.349437 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:21.350438 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:21.350757 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:21.350951 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:21.351226 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:13:21.351377 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:13:21.351436 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:21.351512 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 13:13:21.351545 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":69,"threads_started":1}
I20260926 13:13:21.353551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 22595
I20260926 13:13:21.359505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 22670
I20260926 13:13:21.364809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 22745
2026-09-26T13:13:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2762 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:13:21 dist-test-slave-2dw5 krb5kdc[22963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:13:21 dist-test-slave-2dw5 krb5kdc[22963](info): set up 2 sockets
Sep 26 13:13:21 dist-test-slave-2dw5 krb5kdc[22963](info): commencing operation
krb5kdc: starting...
W20260926 13:13:22.323035 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/test-user.keytab.
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:13:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:13:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:13:22.441957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34667
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34667,127.0.98.61:36727,127.0.98.60:36657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:13:22.521989 22976 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:13:22.522224 22976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:22.522250 22976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:22.522276 22976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:22.523566 22976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:22.523622 22976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:22.523639 22976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:22.523655 22976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:22.523671 22976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:22.525323 22976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34667,127.0.98.61:36727,127.0.98.60:36657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:22.525573 22976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:22.525810 22976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:22.530961 22981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:22.530988 22984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:22.530992 22976 server_base.cc:1034] running on GCE node
W20260926 13:13:22.530962 22982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:22.531354 22976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:22.531687 22976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:22.532877 22976 hybrid_clock.cc:648] HybridClock initialized: now 1790428402532855 us; error 33 us; skew 500 ppm
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:22.535511 22976 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:13:22.535940 22976 webserver.cc:460] Webserver started at http://127.0.98.62:42609/ using document root <none> and password file <none>
I20260926 13:13:22.536281 22976 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:22.536331 22976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:22.536478 22976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:22.537516 22976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/data/instance:
uuid: "2c70f21ad89a446ba1a9895af1dfc570"
format_stamp: "Formatted at 2026-09-26 13:13:22 on dist-test-slave-2dw5"
server_key: "03e4bd022b09280dae6a3f6a0f7b4e01"
server_key_iv: "009c8dafab0d6e7ac516448dc1537b28"
server_key_version: "encryptionkey@0"
I20260926 13:13:22.537839 22976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c70f21ad89a446ba1a9895af1dfc570"
format_stamp: "Formatted at 2026-09-26 13:13:22 on dist-test-slave-2dw5"
server_key: "03e4bd022b09280dae6a3f6a0f7b4e01"
server_key_iv: "009c8dafab0d6e7ac516448dc1537b28"
server_key_version: "encryptionkey@0"
I20260926 13:13:22.539247 22976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:13:22.539986 22991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:22.540132 22976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:13:22.540263 22976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/wal
uuid: "2c70f21ad89a446ba1a9895af1dfc570"
format_stamp: "Formatted at 2026-09-26 13:13:22 on dist-test-slave-2dw5"
server_key: "03e4bd022b09280dae6a3f6a0f7b4e01"
server_key_iv: "009c8dafab0d6e7ac516448dc1537b28"
server_key_version: "encryptionkey@0"
I20260926 13:13:22.540328 22976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:22.559660 22976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:22.561452 22976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:22.561583 22976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:22.565362 22976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34667
I20260926 13:13:22.565399 23043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34667 every 8 connection(s)
I20260926 13:13:22.565734 22976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:13:22.566696 23044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:22.567090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 22976
I20260926 13:13:22.567175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-0/wal/instance
I20260926 13:13:22.567363 392 external_mini_cluster.cc:1428] Setting key 29ce972801230227844015402551642b
I20260926 13:13:22.568758 23044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:13:22.582216 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.569612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.0.98.62:34667)
0926 13:13:22.569751 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.569755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.569795 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:13:22.570289 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.570291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.570314 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.570423 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.571222 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.572668 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.573588 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.574074 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.574197 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.578937 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.578948 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.578948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.578956 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.580485 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.580966 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.580969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.580969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.581057 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.581296 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.581297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.581297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.581560 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.581632 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.581863 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.582009 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260926 13:13:22.582232 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.569430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34667 (local address 127.0.0.1:34818)
0926 13:13:22.569596 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.569642 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:22.570112 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:22.570500 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:22.570503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:22.570868 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:22.571070 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.571076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.572789 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.572791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.573497 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.573500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.574184 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.574186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.574240 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.574776 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:22.574784 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:22.578814 (+ 4030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:22.580570 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.580578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.580580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.580832 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.581124 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.581124 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.581125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.581230 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.581634 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:22.581640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:22.581734 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 13:13:22.581839 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 13:13:22.582107 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":66,"thread_start_us":70,"threads_started":1}
I20260926 13:13:22.583107 23044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:13:22.583467 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.583327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36727 (local address 127.0.0.1:40338)
0926 13:13:22.583375 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.583433 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 13:13:22.583635 22992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36727: connect: Connection refused (error 111)
I20260926 13:13:22.584036 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.583940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36727 (local address 127.0.0.1:40348)
0926 13:13:22.583983 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.584014 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 13:13:22.584136 23044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36727: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36727: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 13:13:22.594054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36727
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34667,127.0.98.61:36727,127.0.98.60:36657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:13:22.633371 23044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } attempt: 1
I20260926 13:13:22.633910 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.633786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36727 (local address 127.0.0.1:40360)
0926 13:13:22.633842 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.633880 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 13:13:22.634341 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.634256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36727 (local address 127.0.0.1:40366)
0926 13:13:22.634297 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.634322 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 13:13:22.634439 23044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36727: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36727: connect: Connection refused (error 111)
W20260926 13:13:22.675261 23051 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:13:22.675508 23051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:22.675532 23051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:22.675561 23051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:22.676805 23051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:22.676846 23051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:22.676865 23051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:22.676880 23051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:22.676896 23051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:22.678510 23051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34667,127.0.98.61:36727,127.0.98.60:36657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:22.678776 23051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:22.679009 23051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:13:22.684108 23059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:22.684126 23056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:22.684082 23057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:22.684335 23051 server_base.cc:1034] running on GCE node
I20260926 13:13:22.684567 23051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:22.684875 23051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:22.686028 23051 hybrid_clock.cc:648] HybridClock initialized: now 1790428402686008 us; error 28 us; skew 500 ppm
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:22.688673 23051 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 13:13:22.689137 23051 webserver.cc:460] Webserver started at http://127.0.98.61:38577/ using document root <none> and password file <none>
I20260926 13:13:22.689348 23051 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:22.689410 23051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:22.689543 23051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:22.690661 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/data/instance:
uuid: "71ce2e42a25043818629576b8050b9dd"
format_stamp: "Formatted at 2026-09-26 13:13:22 on dist-test-slave-2dw5"
server_key: "213aade716ca226ef74d7e16dad918ba"
server_key_iv: "2fe7b73973cf7ca5bdfd93847ecf431d"
server_key_version: "encryptionkey@0"
I20260926 13:13:22.691048 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/wal/instance:
uuid: "71ce2e42a25043818629576b8050b9dd"
format_stamp: "Formatted at 2026-09-26 13:13:22 on dist-test-slave-2dw5"
server_key: "213aade716ca226ef74d7e16dad918ba"
server_key_iv: "2fe7b73973cf7ca5bdfd93847ecf431d"
server_key_version: "encryptionkey@0"
I20260926 13:13:22.692451 23051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:13:22.693162 23066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:22.693327 23051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:13:22.693424 23051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/wal
uuid: "71ce2e42a25043818629576b8050b9dd"
format_stamp: "Formatted at 2026-09-26 13:13:22 on dist-test-slave-2dw5"
server_key: "213aade716ca226ef74d7e16dad918ba"
server_key_iv: "2fe7b73973cf7ca5bdfd93847ecf431d"
server_key_version: "encryptionkey@0"
I20260926 13:13:22.693498 23051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:22.702594 23044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } attempt: 2
I20260926 13:13:22.703013 23051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:22.703116 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.702957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36727 (local address 127.0.0.1:40372)
0926 13:13:22.703028 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.703080 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 13:13:22.703537 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.703407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36727 (local address 127.0.0.1:40378)
0926 13:13:22.703473 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.703507 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 13:13:22.703658 23044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36727: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36727: connect: Connection refused (error 111)
I20260926 13:13:22.705109 23051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:22.705250 23051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:22.709451 23051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36727
I20260926 13:13:22.709529 23118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36727 every 8 connection(s)
I20260926 13:13:22.709872 23051 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/data/info.pb
I20260926 13:13:22.710930 23119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:22.712544 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:13:22.718737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 23051
I20260926 13:13:22.718824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-1/wal/instance
I20260926 13:13:22.719022 392 external_mini_cluster.cc:1428] Setting key 0b1087cd3ce00844dd67543cf0f33290
I20260926 13:13:22.721402 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.713256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34824 (local address 127.0.98.62:34667)
0926 13:13:22.713336 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.713338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.713486 (+ 148us) server_negotiation.cc:407] Connection header received
0926 13:13:22.714094 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.714095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.714109 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.714175 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.714774 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.715304 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.716337 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.716524 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.716570 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.718655 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.718657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.718657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.718665 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.720102 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.720496 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.720496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.720497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.720559 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.720767 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.720767 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.720768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.720909 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.720983 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.721231 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.721314 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:13:22.721513 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.713195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34667 (local address 127.0.0.1:34824)
0926 13:13:22.713365 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.713408 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:22.713996 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:22.714238 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:22.714242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:22.714498 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:22.714695 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.714698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.715395 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.715397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.716270 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.716272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.716602 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.716603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.716712 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.717152 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:22.717161 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:22.718525 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:22.720204 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.720213 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.720215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.720435 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.720614 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.720615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.720615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.720715 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.720963 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:22.720966 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:22.721090 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 13:13:22.721199 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 13:13:22.721328 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260926 13:13:22.722131 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 13:13:22.729084 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.722363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36727 (local address 127.0.0.1:40382)
0926 13:13:22.722409 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.722420 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:22.722460 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:22.722702 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:22.722702 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:22.722823 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:22.722901 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.722903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.723717 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.723718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.724214 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.724215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.724661 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.724662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.724722 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.725113 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:22.725117 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:22.726293 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:22.728161 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.728162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.728162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.728319 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.728485 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.728485 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.728486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.728548 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.728826 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:22.728827 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:22.728878 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 13:13:22.728934 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 13:13:22.729013 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 13:13:22.729264 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.722433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.0.98.61:36727)
0926 13:13:22.722571 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.722576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.722592 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:13:22.722611 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.722613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.722633 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.722690 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.722962 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.723648 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.724283 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.724598 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.724664 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.726431 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.726441 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.726441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.726449 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.728061 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.728379 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.728381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.728382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.728435 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.728605 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.728606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.728606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.728766 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.728829 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.729088 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.729165 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260926 13:13:22.729818 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:13:22.730177 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.730044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56152)
0926 13:13:22.730094 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.730151 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 13:13:22.730312 23068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
I20260926 13:13:22.730765 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.730681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56162)
0926 13:13:22.730726 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.730749 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 13:13:22.730882 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36657: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 13:13:22.745227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasky34iiX/build/release/bin/kudu
/tmp/dist-test-tasky34iiX/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36657
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34667,127.0.98.61:36727,127.0.98.60:36657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:13:22.780125 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } attempt: 1
I20260926 13:13:22.780618 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.780490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56172)
0926 13:13:22.780551 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.780588 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 13:13:22.780997 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.780846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56186)
0926 13:13:22.780914 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.780962 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 13:13:22.781141 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36657: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
I20260926 13:13:22.794834 23044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } attempt: 3
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:13:22.802821 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.795266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36727 (local address 127.0.0.1:40390)
0926 13:13:22.795331 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.795345 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:22.795410 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:22.795546 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:22.795547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:22.795739 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:22.795823 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.795825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.796867 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.796868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.797541 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.797544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.798076 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.798077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.798133 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.798573 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:22.798578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:22.800064 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:22.801732 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.801733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.801734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.801896 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.802171 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.802172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.802172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.802245 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.802530 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:22.802531 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:22.802581 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 13:13:22.802650 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:13:22.802740 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 13:13:22.802904 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.795347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.0.98.61:36727)
0926 13:13:22.795408 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.795411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.795421 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:13:22.795469 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.795470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.795481 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.795529 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.795973 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.796742 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.797624 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.797950 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.798022 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.800157 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.800159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.800159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.800168 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.801643 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.801972 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.801974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.801974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.802034 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.802311 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.802312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.802312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.802446 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.802513 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.802688 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.802775 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 13:13:22.803362 23044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:13:22.803689 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.803589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56200)
0926 13:13:22.803643 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.803668 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 13:13:22.804051 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.803917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56206)
0926 13:13:22.804012 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.804035 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260926 13:13:22.804150 23044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36657: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
W20260926 13:13:22.829504 23126 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:13:22.829730 23126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:13:22.829748 23126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:13:22.829771 23126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:13:22.830986 23126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:13:22.831032 23126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:13:22.831045 23126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:13:22.831058 23126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:13:22.831067 23126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:13:22.832564 23126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34667,127.0.98.61:36727,127.0.98.60:36657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:13:22.832767 23126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:13:22.832952 23126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:13:22.835278 23044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } attempt: 1
I20260926 13:13:22.835757 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.835585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56218)
0926 13:13:22.835673 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.835724 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 13:13:22.836115 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.836022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56228)
0926 13:13:22.836064 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.836094 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 13:13:22.836222 23044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36657: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
W20260926 13:13:22.837965 23131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:13:22.838020 23134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:22.838227 23126 server_base.cc:1034] running on GCE node
W20260926 13:13:22.837937 23132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:13:22.838555 23126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:13:22.838866 23126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:13:22.840055 23126 hybrid_clock.cc:648] HybridClock initialized: now 1790428402840047 us; error 34 us; skew 500 ppm
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:13:22.842531 23126 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 13:13:22.843016 23126 webserver.cc:460] Webserver started at http://127.0.98.60:42639/ using document root <none> and password file <none>
I20260926 13:13:22.843219 23126 fs_manager.cc:359] Metadata directory not provided
I20260926 13:13:22.843279 23126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:13:22.843379 23126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:13:22.844470 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/data/instance:
uuid: "d15e4548a2774cb3b00a45d5d129f9bb"
format_stamp: "Formatted at 2026-09-26 13:13:22 on dist-test-slave-2dw5"
server_key: "42e1edf31d8df95c37cad648e8010894"
server_key_iv: "625bf7234f758a3c7fa04759854b4f49"
server_key_version: "encryptionkey@0"
I20260926 13:13:22.844763 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/wal/instance:
uuid: "d15e4548a2774cb3b00a45d5d129f9bb"
format_stamp: "Formatted at 2026-09-26 13:13:22 on dist-test-slave-2dw5"
server_key: "42e1edf31d8df95c37cad648e8010894"
server_key_iv: "625bf7234f758a3c7fa04759854b4f49"
server_key_version: "encryptionkey@0"
I20260926 13:13:22.846158 23126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:13:22.846949 23141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:13:22.847127 23126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:13:22.847224 23126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/wal
uuid: "d15e4548a2774cb3b00a45d5d129f9bb"
format_stamp: "Formatted at 2026-09-26 13:13:22 on dist-test-slave-2dw5"
server_key: "42e1edf31d8df95c37cad648e8010894"
server_key_iv: "625bf7234f758a3c7fa04759854b4f49"
server_key_version: "encryptionkey@0"
I20260926 13:13:22.847296 23126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:13:22.849292 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } attempt: 2
I20260926 13:13:22.849768 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.849624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56230)
0926 13:13:22.849688 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.849733 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 13:13:22.850127 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.850038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56234)
0926 13:13:22.850076 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.850106 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 13:13:22.850260 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36657: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36657: connect: Connection refused (error 111)
I20260926 13:13:22.858520 23126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:13:22.860718 23126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:13:22.860836 23126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:13:22.864660 23126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36657
I20260926 13:13:22.864699 23193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36657 every 8 connection(s)
I20260926 13:13:22.865023 23126 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/data/info.pb
I20260926 13:13:22.866020 23194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:13:22.867638 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:13:22.870103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasky34iiX/build/release/bin/kudu as pid 23126
I20260926 13:13:22.870213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasky34iiX/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428136114168-392-0/minicluster-data/master-2/wal/instance
I20260926 13:13:22.870422 392 external_mini_cluster.cc:1428] Setting key 68cbc7d937a7d3761de0fc62c22b22be
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:13:22.880313 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.868391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34834 (local address 127.0.98.62:34667)
0926 13:13:22.868512 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.868514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.868684 (+ 170us) server_negotiation.cc:407] Connection header received
0926 13:13:22.869190 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.869191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.869203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.869252 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.869867 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.870532 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.873640 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.873963 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.874052 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.877033 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.877035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.877035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.877043 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.878786 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.879286 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.879288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.879288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.879344 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.879636 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.879636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.879637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.879746 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.879802 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.880137 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.880220 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 13:13:22.880427 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.868322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34667 (local address 127.0.0.1:34834)
0926 13:13:22.868566 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.868603 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:22.869088 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:22.869319 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:22.869322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:22.869574 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:22.869773 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.869777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.870641 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.870642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.873512 (+ 2870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.873516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.874105 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.874107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.874236 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.874793 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:22.874803 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:22.876857 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:22.878886 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.878895 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.878897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.879199 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.879416 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.879417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.879417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.879569 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.879806 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:22.879820 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:22.879962 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 13:13:22.880051 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:13:22.880237 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":99,"threads_started":1}
I20260926 13:13:22.881198 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:13:22.882664 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.870863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.0.98.60:36657)
0926 13:13:22.873642 (+ 2779us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.873647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.873670 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:13:22.873693 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.873695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.873714 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.873777 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.874624 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.875684 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.877026 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.877447 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.877520 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.879806 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.879814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.879814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.879821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.881500 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.881872 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.881874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.881875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.881933 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.882175 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.882176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.882176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.882302 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.882396 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.882515 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.882589 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2741,"thread_start_us":186,"threads_started":1}
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:13:22.888149 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.881461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36727 (local address 127.0.0.1:40406)
0926 13:13:22.881592 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.881603 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:22.881644 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:22.881761 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:22.881762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:22.881875 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:22.881933 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.881935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.882776 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.882778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.883490 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.883492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.883823 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.883824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.883899 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.884245 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:22.884250 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:22.885367 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:22.887150 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.887151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.887151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.887357 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.887569 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.887570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.887571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.887628 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.887882 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:22.887883 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:22.887924 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 13:13:22.887960 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 13:13:22.888048 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 13:13:22.888161 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.881530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.0.98.61:36727)
0926 13:13:22.881591 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.881593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.881633 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:13:22.881686 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.881686 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.881698 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.881760 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.882025 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.882686 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.883565 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.883769 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.883821 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.885474 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.885476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.885476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.885483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.887064 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.887440 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.887441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.887442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.887510 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.887680 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.887681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.887682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.887816 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.887989 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.888031 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.888099 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:13:22.888717 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:13:22.890964 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.883454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34844 (local address 127.0.98.62:34667)
0926 13:13:22.883497 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.883499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.883508 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:13:22.883538 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.883539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.883548 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.883595 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.884009 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.884453 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.885086 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.885358 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.885428 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.887796 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.887798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.887798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.887805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.889616 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.890025 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.890026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.890026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.890098 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.890294 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.890295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.890295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.890460 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.890681 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.890725 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.890828 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:13:22.895705 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.888953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56256)
0926 13:13:22.889162 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.889171 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:22.889214 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:22.889313 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:22.889314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:22.889444 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:22.889476 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.889477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.890238 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.890239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.890969 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.890973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.891445 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.891445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.891492 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.891849 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:22.891854 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:22.893262 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:22.894751 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.894752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.894752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.894974 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.895157 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.895157 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.895158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.895209 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.895451 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:22.895451 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:22.895485 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 13:13:22.895528 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 13:13:22.895601 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260926 13:13:22.895787 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.889009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.0.98.60:36657)
0926 13:13:22.889046 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.889048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.889220 (+ 172us) server_negotiation.cc:407] Connection header received
0926 13:13:22.889259 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.889259 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.889271 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.889313 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.889534 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.890149 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.891049 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.891339 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.891398 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.893349 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.893351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.893351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.893358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.894675 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.895035 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.895037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.895037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.895091 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.895255 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.895256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.895256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.895387 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.895453 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.895569 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.895665 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":18}
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:13:22.897467 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb: Bootstrap starting.
I20260926 13:13:22.898314 23194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:22.898630 23194 log.cc:824] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:22.899253 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.891923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40418 (local address 127.0.98.61:36727)
0926 13:13:22.891985 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.891987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.891995 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:13:22.892018 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.892019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.892029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.892086 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.892612 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.893094 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.894176 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.894366 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.894417 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.896812 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.896813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.896813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.896821 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.898081 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.898468 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.898469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.898469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.898519 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.898700 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.898700 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.898701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.898819 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.899019 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.899092 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.899171 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:13:22.899580 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb: No bootstrap required, opened a new log
I20260926 13:13:22.900600 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:13:22.901331 23194 raft_consensus.cc:348] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } }
I20260926 13:13:22.901473 23194 raft_consensus.cc:729] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d15e4548a2774cb3b00a45d5d129f9bb, State: Initialized, Role: FOLLOWER
I20260926 13:13:22.901559 23194 consensus_queue.cc:260] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } }
I20260926 13:13:22.901885 23194 sys_catalog.cc:564] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:22.901834 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } } }
I20260926 13:13:22.901970 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:22.901541 23018 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2c70f21ad89a446ba1a9895af1dfc570"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34844
W20260926 13:13:22.903607 23211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:13:22.903661 23211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:13:22.903908 23126 master_runner.cc:186] Error getting master registration for 127.0.98.62:34667: OK, instance_id { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" instance_seqno: 1790428402564284 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:13:22.906705 23126 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:13:22.911373 23044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } attempt: 2
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:13:22.919046 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.911779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.0.98.60:36657)
0926 13:13:22.911825 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.911827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.911839 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:13:22.911902 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.911903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.911912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.911984 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.912301 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.913033 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.914030 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.914335 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.914392 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.916586 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.916587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.916587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.916595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.917911 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.918260 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.918261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.918262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.918310 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.918480 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.918480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.918481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.918632 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.918691 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.918865 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.918945 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:13:22.919046 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.911721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56262)
0926 13:13:22.911778 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.911790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:22.911831 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:22.911998 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:22.911999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:22.912181 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:22.912236 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.912237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.913114 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.913116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.913951 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.913954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.914399 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.914400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.914459 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.914956 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:22.914962 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:22.916484 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:22.918000 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.918001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.918002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.918195 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.918365 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.918365 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.918366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.918430 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.918694 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:22.918696 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:22.918746 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 13:13:22.918807 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 13:13:22.918945 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 13:13:22.920874 23044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570: Bootstrap starting.
I20260926 13:13:22.921617 23044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:22.921950 23044 log.cc:824] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:22.922804 23044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570: No bootstrap required, opened a new log
I20260926 13:13:22.924350 23044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } }
I20260926 13:13:22.924507 23044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c70f21ad89a446ba1a9895af1dfc570, State: Initialized, Role: FOLLOWER
I20260926 13:13:22.924590 23044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } }
I20260926 13:13:22.924875 23214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } } }
I20260926 13:13:22.924971 23214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:22.924932 23044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:22.924945 23018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 13:13:22.925171 23018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:22.925894 23018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } }
I20260926 13:13:22.926231 23018 leader_election.cc:288] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [CANDIDATE]: Term 1 election: Requested vote from peers 71ce2e42a25043818629576b8050b9dd (127.0.98.61:36727), d15e4548a2774cb3b00a45d5d129f9bb (127.0.98.60:36657)
I20260926 13:13:22.926539 23168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c70f21ad89a446ba1a9895af1dfc570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d15e4548a2774cb3b00a45d5d129f9bb"
I20260926 13:13:22.926700 23168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [term 0 FOLLOWER]: Advancing to term 1
W20260926 13:13:22.927003 23225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:13:22.927045 23225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:13:22.927661 23168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c70f21ad89a446ba1a9895af1dfc570 in term 1.
I20260926 13:13:22.929174 22993 leader_election.cc:302] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c70f21ad89a446ba1a9895af1dfc570, d15e4548a2774cb3b00a45d5d129f9bb; no voters:
I20260926 13:13:22.929289 23214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:13:22.929436 23214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [term 1 LEADER]: Becoming Leader. State: Replica: 2c70f21ad89a446ba1a9895af1dfc570, State: Running, Role: LEADER
I20260926 13:13:22.929522 23214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } }
I20260926 13:13:22.927649 22976 master_runner.cc:186] Error getting master registration for 127.0.98.61:36727: OK, instance_id { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" instance_seqno: 1790428402708210 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:13:22.929908 22976 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:13:22.929966 23214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c70f21ad89a446ba1a9895af1dfc570. Latest consensus state: current_term: 1 leader_uuid: "2c70f21ad89a446ba1a9895af1dfc570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } } }
I20260926 13:13:22.930063 23214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:22.930249 23233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:13:22.930442 23233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:13:22.932029 23168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [term 1 FOLLOWER]: Refusing update from remote peer 2c70f21ad89a446ba1a9895af1dfc570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:13:22.932461 23231 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:13:22.933849 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c70f21ad89a446ba1a9895af1dfc570. Latest consensus state: current_term: 1 leader_uuid: "2c70f21ad89a446ba1a9895af1dfc570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } } }
I20260926 13:13:22.933949 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:22.940418 23214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c70f21ad89a446ba1a9895af1dfc570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } } }
I20260926 13:13:22.940531 23214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:22.941023 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c70f21ad89a446ba1a9895af1dfc570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } } }
I20260926 13:13:22.941111 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:22.941401 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } attempt: 3
I20260926 13:13:22.941462 23234 mvcc.cc:205] Tried to move back new op lower bound from 7333594738407243776 to 7333594738400653312. Current Snapshot: MvccSnapshot[applied={T|T < 7333594738407243776}]
I20260926 13:13:22.941872 23238 mvcc.cc:205] Tried to move back new op lower bound from 7333594738407243776 to 7333594738400653312. Current Snapshot: MvccSnapshot[applied={T|T < 7333594738407243776}]
I20260926 13:13:22.944263 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.931704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.0.98.62:34667)
0926 13:13:22.931745 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.931747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.933628 (+ 1881us) server_negotiation.cc:407] Connection header received
0926 13:13:22.933791 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.933792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.933816 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.933844 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.935270 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.936014 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.937596 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.937898 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.937943 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.940367 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.940369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.940370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.940377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.942432 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.943157 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.943158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.943158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.943230 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.943547 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.943547 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.943548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.943673 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.943772 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.944093 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.944175 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:13:22.947395 23233 catalog_manager.cc:1347] Generated new cluster ID: f7055195464243d78ea88e401f2a5deb
I20260926 13:13:22.947449 23233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:13:22 dist-test-slave-2dw5 krb5kdc[22963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:13:22.952600 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.933391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.0.98.61:36727)
0926 13:13:22.933436 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.933438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.945025 (+ 11587us) server_negotiation.cc:407] Connection header received
0926 13:13:22.945129 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.945130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.945141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.945182 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.946657 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.947348 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.948279 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.948573 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.948632 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.949850 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.949852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.949852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.949859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.951443 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.951813 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.951814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.951815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.951862 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.952035 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.952035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.952035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.952154 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.952229 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.952396 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.952500 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:13:22.953670 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.941660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36657 (local address 127.0.0.1:56280)
0926 13:13:22.941714 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 13:13:22.941726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:13:22.941774 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:13:22.942103 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:13:22.942104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:13:22.942286 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:13:22.942346 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.942348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.943910 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.943911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.945192 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:13:22.945195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.945814 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.945816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:13:22.945885 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.946568 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:13:22.946573 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:13:22.948401 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:13:22.952256 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.952257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.952257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.952756 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.953097 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:13:22.953098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:13:22.953098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:13:22.953157 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:13:22.953405 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:13:22.953406 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:13:22.953446 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 13:13:22.953488 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 13:13:22.953579 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 13:13:22.954108 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.941749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.0.98.60:36657)
0926 13:13:22.941896 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.941898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.941929 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:13:22.941949 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.941950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.941960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.942433 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.942506 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.943775 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.945337 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.945741 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.945798 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.948505 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.948506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.948507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.948513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.952106 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.952967 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.952969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.952969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.953029 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.953217 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.953217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.953218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.953334 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.953774 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.953815 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.954006 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260926 13:13:22.955475 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd: Bootstrap starting.
I20260926 13:13:22.956159 23119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd: Neither blocks nor log segments found. Creating new log.
I20260926 13:13:22.956478 23119 log.cc:824] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd: Log is configured to *not* fsync() on all Append() calls
I20260926 13:13:22.956755 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:22.932956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.0.98.60:36657)
0926 13:13:22.932995 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:22.932996 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:22.941934 (+ 8938us) server_negotiation.cc:407] Connection header received
0926 13:13:22.942029 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:22.942030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:22.942040 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:22.942081 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:22.942731 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.943415 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.944329 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:22.944613 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:22.944652 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:22.946126 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:13:22.946127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:13:22.946127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:13:22.946132 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:13:22.947583 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.948575 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.948576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.948577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.948681 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:13:22.956065 (+ 7384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:13:22.956067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:13:22.956067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:13:22.956202 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:13:22.956281 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:22.956509 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 13:13:22.956670 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:13:22.957624 23233 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:13:22.958361 23233 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:13:22.963768 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd: No bootstrap required, opened a new log
I20260926 13:13:22.965835 23119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } }
I20260926 13:13:22.965994 23119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71ce2e42a25043818629576b8050b9dd, State: Initialized, Role: FOLLOWER
I20260926 13:13:22.966090 23119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } }
I20260926 13:13:22.966454 23119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:13:22.966926 23092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:13:22.967851 23092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [term 1 FOLLOWER]: Refusing update from remote peer 2c70f21ad89a446ba1a9895af1dfc570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:13:22.968204 23093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2c70f21ad89a446ba1a9895af1dfc570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "71ce2e42a25043818629576b8050b9dd"
I20260926 13:13:22.969003 23093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c70f21ad89a446ba1a9895af1dfc570 in term 1.
I20260926 13:13:22.969321 23241 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 13:13:22.972565 23260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:13:22.972631 23260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:13:22.972769 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c70f21ad89a446ba1a9895af1dfc570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } } }
I20260926 13:13:22.972838 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:22.972879 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c70f21ad89a446ba1a9895af1dfc570. Latest consensus state: current_term: 1 leader_uuid: "2c70f21ad89a446ba1a9895af1dfc570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } } }
I20260926 13:13:22.972922 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:22.972947 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c70f21ad89a446ba1a9895af1dfc570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } } }
I20260926 13:13:22.972980 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:13:22.974095 23247 mvcc.cc:205] Tried to move back new op lower bound from 7333594738504826880 to 7333594738400653312. Current Snapshot: MvccSnapshot[applied={T|T < 7333594738407243776}]
I20260926 13:13:22.978798 23241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c70f21ad89a446ba1a9895af1dfc570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c70f21ad89a446ba1a9895af1dfc570" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34667 } } peers { permanent_uuid: "71ce2e42a25043818629576b8050b9dd" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36727 } } peers { permanent_uuid: "d15e4548a2774cb3b00a45d5d129f9bb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36657 } } }
I20260926 13:13:22.978964 23241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570 [sys.catalog]: This master's current role is: LEADER
I20260926 13:13:22.980135 23233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c70f21ad89a446ba1a9895af1dfc570: Generated new TSK 0
I20260926 13:13:22.980357 23233 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:13:23.004115 23048 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.002685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.0.98.62:34667)
0926 13:13:23.002730 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.002732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.003387 (+ 655us) server_negotiation.cc:407] Connection header received
0926 13:13:23.003556 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.003557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.003568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.003599 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.004044 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34862: BlockingRecv error: recv got EOF from 127.0.0.1:34862 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 13:13:23.005035 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.003080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.0.98.60:36657)
0926 13:13:23.003119 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.003121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.004535 (+ 1414us) server_negotiation.cc:407] Connection header received
0926 13:13:23.004629 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.004630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.004640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.004669 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.004977 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56296: BlockingRecv error: recv got EOF from 127.0.0.1:56296 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 13:13:23.006074 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.005544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.0.98.61:36727)
0926 13:13:23.005588 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.005590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.005599 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:13:23.005625 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.005626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.005636 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.005712 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.006002 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40430: BlockingRecv error: recv got EOF from 127.0.0.1:40430 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 13:13:23.008206 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.007505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.0.98.61:36727)
0926 13:13:23.007551 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.007553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.007660 (+ 107us) server_negotiation.cc:407] Connection header received
0926 13:13:23.007772 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.007773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.007793 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.007822 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.008160 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40434: BlockingRecv error: recv got EOF from 127.0.0.1:40434 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 13:13:23.011435 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.010584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.0.98.61:36727)
0926 13:13:23.010619 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.010621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.010887 (+ 266us) server_negotiation.cc:407] Connection header received
0926 13:13:23.010988 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.010989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.011000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.011030 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.011382 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40450: BlockingRecv error: recv got EOF from 127.0.0.1:40450 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 13:13:23.015653 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.014807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.0.98.61:36727)
0926 13:13:23.014871 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.014873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.015088 (+ 215us) server_negotiation.cc:407] Connection header received
0926 13:13:23.015188 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.015190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.015201 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.015231 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.015599 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40464: BlockingRecv error: recv got EOF from 127.0.0.1:40464 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 13:13:23.022095 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.021083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.0.98.61:36727)
0926 13:13:23.021126 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.021128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.021469 (+ 341us) server_negotiation.cc:407] Connection header received
0926 13:13:23.021593 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.021594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.021606 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.021635 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.022036 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40480: BlockingRecv error: recv got EOF from 127.0.0.1:40480 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 13:13:23.032647 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.031728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.0.98.61:36727)
0926 13:13:23.031774 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.031777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.032033 (+ 256us) server_negotiation.cc:407] Connection header received
0926 13:13:23.032137 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.032138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.032150 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.032179 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.032584 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40486: BlockingRecv error: recv got EOF from 127.0.0.1:40486 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 13:13:23.051280 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.050271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.0.98.61:36727)
0926 13:13:23.050329 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.050330 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.050594 (+ 264us) server_negotiation.cc:407] Connection header received
0926 13:13:23.050695 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.050696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.050709 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.050738 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.051195 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40488: BlockingRecv error: recv got EOF from 127.0.0.1:40488 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 13:13:23.085510 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.084816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.0.98.61:36727)
0926 13:13:23.084863 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.084865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.084994 (+ 129us) server_negotiation.cc:407] Connection header received
0926 13:13:23.085049 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.085050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.085063 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.085108 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.085443 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40502: BlockingRecv error: recv got EOF from 127.0.0.1:40502 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 13:13:23.151209 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.150550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.0.98.61:36727)
0926 13:13:23.150605 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.150607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.150708 (+ 101us) server_negotiation.cc:407] Connection header received
0926 13:13:23.150756 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.150757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.150768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.150813 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.151145 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40514: BlockingRecv error: recv got EOF from 127.0.0.1:40514 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 13:13:23.281100 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.280336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.0.98.61:36727)
0926 13:13:23.280393 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.280394 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.280422 (+ 28us) server_negotiation.cc:407] Connection header received
0926 13:13:23.280516 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.280517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.280532 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.280634 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.281035 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40528: BlockingRecv error: recv got EOF from 127.0.0.1:40528 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 13:13:23.538968 23123 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:13:23.538254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.0.98.61:36727)
0926 13:13:23.538328 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:23.538329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:23.538409 (+ 80us) server_negotiation.cc:407] Connection header received
0926 13:13:23.538474 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:23.538475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:23.538488 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:23.538536 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:13:23.538907 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40534: BlockingRecv error: recv got EOF from 127.0.0.1:40534 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:13:23.904102 23211 catalog_manager.cc:1259] Loaded cluster ID: f7055195464243d78ea88e401f2a5deb
I20260926 13:13:23.904181 23211 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb: loading cluster ID for follower catalog manager: success
I20260926 13:13:23.905085 23211 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb: acquiring CA information for follower catalog manager: success
I20260926 13:13:23.905395 23211 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d15e4548a2774cb3b00a45d5d129f9bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:13:23.973044 23260 catalog_manager.cc:1259] Loaded cluster ID: f7055195464243d78ea88e401f2a5deb
I20260926 13:13:23.973150 23260 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd: loading cluster ID for follower catalog manager: success
I20260926 13:13:23.974005 23260 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd: acquiring CA information for follower catalog manager: success
I20260926 13:13:23.974305 23260 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 71ce2e42a25043818629576b8050b9dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:13:24.054386 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:24.052083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.0.98.61:36727)
0926 13:13:24.052177 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:24.052179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:24.052188 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:13:24.052259 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:24.052260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:24.052308 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:24.052373 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:13:24.052477 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:24.052993 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:24.053769 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:24.053963 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:24.054022 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:24.054116 (+ 94us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:13:24.054218 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:13:24.054269 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:24.054297 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 13:13:24.054314 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":47,"threads_started":1}
I20260926 13:13:24.058284 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:24.056137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.0.98.60:36657)
0926 13:13:24.056260 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:24.056262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:24.056285 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:13:24.056346 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:24.056347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:24.056382 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:24.056457 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:13:24.056534 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:24.056996 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:24.057654 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:24.057847 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:24.057899 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:24.058002 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:13:24.058105 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:13:24.058155 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:24.058171 (+ 16us) server_negotiation.cc:299] Negotiation successful
0926 13:13:24.058188 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":50,"threads_started":1}
I20260926 13:13:24.062645 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:24.059795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.0.98.61:36727)
0926 13:13:24.059836 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:24.059838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:24.060046 (+ 208us) server_negotiation.cc:407] Connection header received
0926 13:13:24.060137 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:24.060138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:24.060147 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:24.060203 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:13:24.060338 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:24.060845 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:24.061922 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:24.062203 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:24.062260 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:24.062400 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:13:24.062431 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:13:24.062458 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:24.062560 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 13:13:24.062583 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:13:24.062638 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:24.060001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.0.98.60:36657)
0926 13:13:24.060036 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:24.060038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:24.060045 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:13:24.060095 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:24.060096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:24.060107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:24.060150 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:13:24.060235 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:24.060940 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:24.061941 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:24.062123 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:24.062174 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:24.062322 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:13:24.062355 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:13:24.062402 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:24.062542 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 13:13:24.062566 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:13:24.062930 23342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:13:24.059773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34870 (local address 127.0.98.62:34667)
0926 13:13:24.060176 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 13:13:24.060179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:13:24.060186 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:13:24.060228 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:13:24.060230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:13:24.060276 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:13:24.060376 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:13:24.060473 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:24.061157 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:24.061976 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:13:24.062268 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:13:24.062315 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:13:24.062510 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:13:24.062620 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:13:24.062674 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:13:24.062805 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 13:13:24.062831 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":67,"threads_started":1}
I20260926 13:13:24.064591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 22976
I20260926 13:13:24.069486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 23051
I20260926 13:13:24.073674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasky34iiX/build/release/bin/kudu with pid 23126
2026-09-26T13:13:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2706 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5469 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (267963 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST